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CHAPTER  1 
INTRODUCTION 

1.1.  Motivation  and  General  Background 

The  present  thesis  deals  with  certain  topics  in  the  area  of  Nash 
and  Leader-Follower  dynamic  games  with  state  information  available  to  the 
players.  Our  motivation  for  this  study  was  the  belief  that  game  theory 
provides  formalisms  and  results  which  are  useful  in  describing, understanding 
and  manipulating  successfully  large  scale  and  hierarchical  engineering 
systems.  Hierarchical  and  large  scale  systems  have  received  considerable 
attention  during  the  last  few  years;  firstly  because  of  their  importance 
in  engineering,  economics  and  other  areas,  and  secondly  because  of  the 
increased  capability  of  computer  facilities  [37],  [38].  An  important 
characteristic  of  many  large  scale  systems  is  the  presence  of  many  decision 
makers  with  different  and  usually  conflicting  goals.  The  existence  of  many 
decision  makers  who  interact  through  the  system  and  have  different  goals  may 
be  an  inherent  property  of  the  system  under  consideration  (e.g.,  a  market 
situation),  or  may  be  simply  the  result  of  modeling  the  system  as  such  (e.x., 
a  large  system  decomposed  to  subsystems  for  calculation  purposes).  Differential 
games  are  useful  in  modeling  and  studying  dynamic  systems  where  more  than 
one  decision  maker  is  involved.  Most  of  the  questions  posed  in  the  area  of 
the  classical  control  problem  may  be  considered  in  a  game  situation,  but 
their  resolution  is  generally  more  difficult.  In  addition,  many  questions 
can  be  posed  in  a  game  framework,  which  are  meaningless  or  trivial  in  a 
classical  control  problem  framework.  The  superior  conceptual  wealth  of  game 
over  control  problems,  which  makes  them  potentially  much  more  applicable, 


counterbalances  the  additional  difficulties  encountered  in  their  solution. 

The  theory  of  games  achieved  its  maturity  as  a  field  of  active 
research  basically  due  to  John  von  Neumann.  The  publication  of  the  book 
"Theory  of  Games  and  Economic  Behavior"  by  J.  V.  Neumann  and  0.  Morgenstem 
[66],  gave  the  impetus  for  research  in  Game  Theory.  Although  the  theory  of 
games  was  initially  appealing  almost  exclusively  to  some  Economists,  it's 
usefulness  in  applications  and  challenge  as  an  area  of  research  is  today 
recognized  by  many  Engineers,  Mathematicians,  Economists,  Sociologists, 
Psychologists  and  Political  Scientists.  The  attention  of  the  researchers 
was  initially  almost  exclusively  focused  on  static  two-player,  zero-sum  games. 
Rufus  Isaacs  by  imposing  a  certain  type  of  structure  to  the  sets  and  the 
functions  describing  a  two-player,  zero-sum  game  _  in  it's  abstract  general 
form  established  the  area  of  two-player,  zero-sum  differential  games,  [67]. 

In  the  literature,  the  characterization  "static"  is  usually  attributed  to  games 
which  are  presented  so  as  to  have  suppressed  in  the  final  formalism  any 
explicit  dependence  on  the  evolution  of  time.  The  characterization  "dynamic" 
is  attributed  to  games  where  evolution  of  time  is  not  suppressed  in  the  final 
formalism  with  which  the  game  is  presented.  The  difference  between  static 
and  dynamic  games  lies  mostly  in  the  way  that  we  choose  to  state  the  game, 
since  every  dynamic  game  can  be  stated  equivalently  as  a  static  one.  (The 
discussion  on  normal  and  extensive  form  of  a  game,  see  [66],  is  pertinent 
here).  Nonetheless  by  adopting  a  static  game  theoritic  description  of  a 
certain  problem,  in  which  problem  the  evolution  of  time  is  present,  we  hide 
a  lot  of  the  underlying  concepts  which  are  related  to  the  evolution  of  time 
and  might  be  useful  in  posing  and  studying  questions  about  the  game  as  well 
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as  in  interpreting  several  theoretical  results.  Dynamic  games  are  usually 
described  by  differential  of  difference  equations  (deterministic  or  stochastic) 
and  the  term  differential  games  is  usually  reserved  for  the  former  description. 
The  present  thesis  deals  with  dynamic  games  with  more  emphasis  on  the  differen¬ 
tial  ones. 

The  two  particular  types  of  games  which  are  of  interest  here  are 
the  so  called  Nash  differential  games  and  Leader-F.ollower  (LF)  differential 
games,  see  [1]  and  [33]  respectively.  The  general  definitions  of  Nash  and 
LF  games  are  as  follows.  Let  U,  V  be  two  sets  and  J^,  J2  two  functions 


J.:  UxV-R,  i  =  1,2 , 
x  ’ 


(1.1) 


J^,  are  referred  to  as  the  cpsts  and  U,  V  as  the  strategy  spaces  of 

< 

players  1  and  2  respectively. 

Definition  of  a  Nash  Equilibrium;  A  pair  (u*,  v*)  fUxV  is  called  a  Nash 
equilibrium  pair  if  (u*,v*)  satisfies 


JL(u*,v*)  £  J^Cu.v*),  V  u  6  U 


J2(u*,v*)  £  J2(u*,v),  VvgV. 


(1.2) 


To  define  an  .LF  equilibrium  pair  we  need  first  to  define  a  mapping  T. 
Consider  the  set  valued  mapping  T 


T:  U  -  V,  u*-  Tu  c  V 


(1.3) 


defined  by 


Tu  =  t.v|v  =  arg  inf[J2(u,v);  v€V]J, 


(1.4) 


Clearly  Tu  *  p  if  the  inf  is  not  achieved.  We  also  consider  the  minimization 
problem 


inf  J  L (u , v) 

subject  to:  u£U,  v£Tu, 


(1.5) 


where  we  use  the  usual  convention  J^(u,v)  =  +  ®  if  v€Tu  =  p. 

Definition  of  an  LF  equilibrium:  A  pair  (u*,v*)  £UxV  is  called  a  Stackel- 
berg  equilibrium  pair  if  (u*,v*)  solves  (1).  (In  LF  games  it  is  standard 
to  say  that  a  leader  chooses  u  6U  and  has  cost  and  a  follower  chooses 
v(V  and  has  cost  Jj). 

If  we  are  interested  in  a  Nash  equilibrium  we  say  that  we  have  a  Nash  game 
and  if  we  are  interested  in  an  LF  equilibrium  we  say  that  we  have  an  LF 
game.  Notice  that  we  do  not  ask  that  =  i«6.  we  are  dealing  with 

nonzero-sum  games. 

Nash  games  provide  a  formalism  for  describing  situations  of  conflict 
where  player  1  chooses  u€U,  player  2  chooses  v(V,  player  1  is  interested 
in  minimizing  his  cost  J^,  player  2  is  interested  in  minimizing  his  cost  J 
and  the  two  players  do  not  trust  each  other  and  do  not  cooperate.  Nonetheless, 
each  one  assumes  for  the  other  that  he  will  act  in  a  rational  way  and  try  to 
satisfy  (1.2).  LF  games  provide  a  formalism  for  describing  the  following 
situation.  The  follower  tries  to  minimize  his  cost  for  a  given  choice  of 
u  e  U  by  the  leader.  The  leader  whose  interest  is  to  minimize  J^,  knowing 
the  follower ' s  rationa  1  and  having  the  privilege  to  choose  his  strategy  first, 
wishes  to  announce  a  u*eU,  such  that  together  with  the  follower's  reaction  v*  to 
u*,  will  result  to  the  minimum  possible  J^.  Notice  that  in  LF  games  the 


J 
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J 

I 

1 

I 

I 

» 

I 

» 
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leader  chooses  and  declares  his  u  first  i,e.  there  is  a  hierarchy  in  the 
decision  making  in  contrast  with  Nash  games  where  the  players  choose  and 
declare  their  choices  simultaneously.  Notice  also  that  in  a  Nash  game  both 
players  must  know  U,V,J^,J2  while  in  an  LF  game  the  leader  must  know  U,V,J^, 

J2  and  the  follower  must  know  u*,V  and  J2. 

Nash  games  were  first  introduced  and  studied  in  a  static  framework 
by  J.  Nash  [68],  The  dynamic  version  of  Nash  games  was  first  introduced  and 
studied  by  Starr  and  Ho  [1].  LF  games  was  first  introduced  by  von  Stackel- 
berg  [26],  who  studied  a  simple  static  game  in  a  .inite  dimensional  framework. 
The  dynamic  version  of  LF  games  was  first  introduced  and  studied  by  Chen  and 
Cruz  and  Simaan  and  Cruz,  [27-29].  The  introduction  of  Nash  and  LF  dynamic 
•  games  was  motivated  significantly  by  R.  Isaacs  work  [67]  on  two-person,  zero- 
sum  dynamic  games. 

In  order  to  describe  Nash  and  LF  two-player  differential  games  we 
consider  the  state  equation 

x(t)  *  f (x(t) ,u(t) ,v(t)  ,t) ,  x(tQ)  =  xq,  tS[tQ,tf]  (1.6) 
and  the  cost  functionals 

J1(u,v)  =  gi(x(tf))  +  £  LiCx(t) ,u(t) ,v(t) ,t)dt,  i=l , 2 ,  (1.7) 

o 

n  _  _  ^9 

where  x(t)€R  ,  u(t)€R  i,  v(t)£R  and  f,  L^,  g^  are  functions  with  appropriate 
domains,  and  ranges  in  Rn,  R^ ,  R^  respectively  and  satisfy  certain  continuity 
and  differentiability  assumptions.  Also  i^U,  v€V  where  U  and  V  are  defined 
below  and  u(t) ,  v(t)  are  the  values  of  u  and  v  respectively,  at  time  t.  At 
each  instant  of  time  t  player  1  has  a  certain  information  about  the  previous 
values  of  the  trajectory  and  the  previous  values  of  his  opponent's  control 


6 


values,  i.e,  about  x^  and  v  ,  More  precisely,  at  time  t,  player  1  knows 
I  (x  ,v  t)  see  (1.9).  Before  player  1  chooses  his  u,  he  only  knows  the 
function  1^ ,  Similarly  player  2  knows  a  function  I  which  means  that  at  time 
t  we  will  know  ^(x^u^jt),  At  the  beginning  of  the  game  the  players  choose 
functions  u  and  v  and  substitute  u(I^ (x^ ,v  , t) ) ,  vCl^ (x^ ,u  ,t) ) ,  in  the 
place  of  u(t) ,  v(t)  in  the  state  equation  and  the  cost  functionals.  The 
differential  equation 

x(t)  =  f(x(t),  u^Cx  ,v  ,t),v(I2(xt,u  .t),t) 

x(to)  =  xq,  te[TQ,tf]  (1.8) 

is  solved  where 


x„  =  (x(t) It  <  x  <  t} 
t  o 

Ut  =  {udlCx^^.T)^)!^  <  T  <  t>  (1.9) 

vt  =  tv(I2^xx»UT,T^’T^lto  **  T  ** 

and  the  u,v  are  fixed.  Assuming  that  the  solution  of  (1.8)  exists  and  is 
unique,  call  it  x(t;u,v)  we  can  calculate  the  values  of  the  cost  functionals 
which  will  depend  on  u  and  v.  The  functions  I^,^,  their  images  and  the 
possible  continuity  dif ferentiabily  and  image  assumptions  on  u  and  v  deter¬ 
mine  U  and  V.  For  example,  if  I^(a,8,Y)  =  (a(tQ),Y),  i=l,2,  i.e,  both 
players  have  no  information  except  knowledge  of  the  current  instant  of  time  t 

and  x  we  can  consider  U,  V  to  be  sets  of  piecewise  continuous  function  of 
o 

time  which  depend  also  on  xq.  This  case  is  called  open-loop.  Another  example 
is  to  choose  1^,  such  that  the  players  know  at  each  instant  of  time  t, 
the  triple  {xo,x(t),t},  Notice  that  in  general,  1^  and  have  function  spaces 
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as  domains  and  images.  For  the  set  up  of  this  paragraph  the  definitions  of 
Nash  and  LF  games  are  those  given  before. 

In  the  case  where  Cl. 6)  is  substituted  by  a  stochastic  differential 
equation  and  expectation  is  taken  of  the  cost  functionals  (1.7),  we  say  that 
we  have  a  stochastic  differential  game.  Also,  we  can  substitute  (1.6)  by  a 
difference  (stochastic  or  not)  equation,  and  consider  summations  (with 
expectation  in  front),  in  place  of  the  integral  cost  functionals  (1,7). 

We  can  pose  several  questions  about  the  differential  game  described 
above.  For  example,  for  given  we  can  study  existence  and  uniqueness 

of  solutions  and  necessary  and  sufficient  conditions  for  a  pair  (u*,v*)  to 
be  an  equilibrium  (Nash  or  LF)  pair.  We  can  also  investigate  the  dependence 
of  an  equilibrium  pair  and  of  the  resulting  costs  on  ^  and  12,  These  and 
many  other  questions  which  have  been  posed  elsewhere  are  difficult  to  study. 
The  case  where  I.(a,£,Y)  =  (cx(to),y),  i-1,2  is  easier  and  there  are  several 
results  about  it.  In  the  present  thesis  we  will  try  to  answer  questions  on 
existence,  uniqueness  and  characterization  by  necessary  and  sufficient  condi¬ 
tions  for  cases  where  are  such  that  provide  information  about  the 

current  and  the  previous  values  of  the  state  to  the  players. 

1.2,  Outline  of  the  Thesis 

In  the  rest  of  the  introduction  we  give  a  general  outline  of  the 
results  of  the  present  work  and  relate  them  to  already  existing  ones. 

In  Chapter  2  we  deal  with  Nash  two-player  differential  games, 
where  (1,6)  is  linear  in  x,  u,  v  and  the  costs  (1.7)  are  quadratic,  i.e. 

linear  quadratic  Nash  games, except  in  the  Section  2.4  where  nonlinear 


8 


f  and  g^,g2>L1,L2  are  also  considered.  The  information  available  to  each 
player  at  time  t  is  x(t)  and  t  and  we  call  this  case  closed  loop,  The  Nash 
solution  for  linear  quadratic  games  has  been  studied  in  several  papers,  see 
[1-10].  Despite  the  many  results  available  in  this  area,  those  concerning 
existence  and  uniqueness  of  optimal  strategies  are  far  from  being  satisfac¬ 
tory.  This  holds  true  especially  if  the  strategies  take  into  account  infor¬ 
mation  about  the  present  and  past  values  of  the  state  of  the  system.  In 
this  context  [6],  [17],  [18]  can  be  pointed  out.  In  these  papers,  the  non¬ 
uniqueness  of  the  Nash  equilibrium  strategies  was  demonstrated  when  the 

current  state  x(t)  and  the  initial  state  x  are  available  to  at  least  one  of 

o 

the  players.  It  was  also  shown  that  in  the  case  of  discrete- time  linear 
quadratic  games,  under  invertibility  conditions  of  certain  matrices, 
if  noise  is  introduced  to  the  state  equation,  then  the  Nash  equilibrium 
strategies  linear  in  the  current  state  x(t)  (assuming  they  exist)  are  the 
unique  solution  without  restricting  a  priori  the  admissible  strategies  to  be 
linear  in  the  current  state.  The  closed  loop  Nash  strategies  are  not  neces¬ 
sarily  linear  [6],  and  even  if  restriction  to  linear  strategies  is  made,  still 
little  is  known  concerning  their  existence,  properties,  interpretation  in 
terms  of  solutions  to  the  coupled  Riccati  equations,  and  the  stability 
of  the  closed  loop  system.  For  the  linear  quadratic  game  over  a  finite 
period  of  time  [0,T],  there  are  certain  existence  results  for  closed  loop 
Nash  strategies,  assuming  that  T  is  sufficiently  small  and/or  that  the  strate¬ 
gies  lie  in  compact  subsets  of  the  admissible  strategy  spaces  [3],  [7],  [8]. 

In  [2],  [5]  the  boundedness  of  the  solutions  of  certain  Riccati  type  differ¬ 
ential  equations  is  assumed  in  order  to  guarantee  the  existence  of  Nash 
strategies,  Finally,  [15]  deals  with  the  static  N-person  Nash  game,  under 


compactness  and  convexity  assumptions  for  the  strategy  spaces  and  concavity 
assumptions  for  the  criteria,  Chapter  2  contains  three  sections,  In  Sec¬ 
tion  2.2  we  deal  with  existence  of  Nash  equilibria  which  are  linear  in  x(t) , 
the  time  interval  is  [0,“)  and  all  the  matrices  involved  in  the  description 
of  the  state  equation  and  cost  functionals  are  constant.  For  this  case  there 
was  no  existence  result  available  known  to  us.  Although  our  results  do  not 
solve  the  problem  completely  they  are  applicable  to  a  subclass  of  problems. 

They  are  stated  in  terms  of  conditions  on  the  norms  of  the  matrices  involved 
and  they  do  not  depend  on  controllability  or  observability  assumptions.  They 
can  be  viewed  as  conditions  for  solution  of  certain  coupled  algebraic  Riccati 
type  matrix  equations.  We  derive  our  basic  existence  result  by  using  Brouwer's 
Fixed  Point  Theorem  in  a  way  which  disengages  our  result  from  a  local  character 
that  a  straightforward  application  of  Brouwer's  Theorem  would  impose.  The 
generalization  of  our  results  to  the  N-player  case  is  obvious.  It  should  be 
pointed  out  that  for  many  of  the  conditions  presented  no  assumptions  of  con¬ 
trollability,  observability  or  semidefiniteness  are  made.  Therefore  we 
actually  single  out  a  region  of  parameter  space  (A.B^ ,Q^ ,R„ )  where  in  the 
existence  of  solutions  does  not  depend  on  controllability  and  observability. 

This  region  is  necessarily  contained  in  the  region  where  A  is  asymptotically 
stable,  or  is  the  neighborhood  of  a  parameter  point  for  which  a  solution  of 
an  auxiliary  control  problem  exists.  Outside  this  region  the  existence  of 
solutions  will  depend  in  general  on  controllability  and  observability  proper¬ 
ties,  but  presently  conditions  under  which  existence  can  be  guaranteed  are 
not  known.  Finally  as  a  byproduct  of  the  use  of  Brouwer's  Theorem  we  in¬ 
terpret  some  existing  results  about  the  algebraic  Riccati  equation  of  the 
control  problem  in  a  new  way,  see  Remark  iii  in  2.1.3.  In  Section  2.2  we 


consider  a  Linear  Quadratic  Nash  Game  over  a  finite  period  of  time  [0,TJ. 
The  matrices  involved  are  piecewise  continuous  functions  of  time,  The 


existence  of  linear  closed  loop  Nash  strategies  depends  on  the  existence  of 
continuous  solutions  to  an  associated  system  of  two  coupled  Riccati  differen¬ 
tial  equations  over  [0,T],  Sufficient  conditions  for  existence  are 
derived  by  using  a  simple  result  from  the  theory  of  differential 
inequalities.  The  conditions  are  given  in  terms  of  upper  bounds  on  the 
length  of  the  time  interval  of  interest  and  do  not  depend  on  controllability 
or  observability  assumptions.  The  positive  (semi-)  definiteness  assumptions 
on  are  not  used  in  proving  the  existence  of  solutions. 

Although  the  conditions  give  only  a  partial  answer  to  the  question 
of  existence  of  solutions,  they  can  nonetheless  provide  a  positive  answer 
for  a  certain  class  of  problems.  The  extension  of  the  present  results  to  the 
N-players  case  is  straightforward. 

In  Section  2.3  we  consider  the  case  where  f ,L^,L2>g^,g2  are 
analytic  functions,  the  time  interval  is  finite  and  fixed  and  the  players  have 
closed  loop  information.  It  is  shown  that  if  the  strategy  spaces  are  re¬ 
stricted  to  analytic  functions  of  the  state  and  time  then  the  Nash  equilibrium 
pair  is  unique  -  if  it  exists,  In  particular,  for  a  linear  quadratic  game, 
where  the  matrices  involved  are  analytic  functions  of  time,  it  is  shown  that 
if  the  coupled  Riccati  differential  equations  have  a  solution,  then  the  Nash 
equilibrium  strategies  which  are  affine  functions  of  the  state  constitute  the 
unique  analytic  solution  pair.  Although  the  result  of  this  section  is  proven 
under  the  strong  analyticity  assumptions,  it  provides  at  least  a  partial 
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answer  to  the  question  of  uniqueness  for  a  certain  class  of  problem.  It  pro¬ 
vides  also  an  additional  characterization  of  Nash  equilibrium  strategies 
which  are  affine  functions  of  the  state  in  the  context  of  linear  quadratic 
games  with  analytic  matrices,  since  it  shows  that  these  strategies  constitute 
the  solution  over  strategy  spaces  much  larger  than  those  which  are  apriori 
restricted  to  be  affine  in  the  state  strategies.  The  introduction  of  analyti- 
city  assumptions  removes  the  nonuniqueness  of  the  Nash  solution  for  determinis¬ 
tic  differential  games  ,  which  is  analogous  to  the  removal  of  nonuniqueness  of 
Nash  solutions  by  introduction  of  noise  {17],  {18],  The  extension  of  these 
results  to  the  N-player  case  is  straightforward. 

In  Chapter  3  we  study  LF  differential  games  where  the  leader  knows 

at  each  instant  of  time  t,  the  values  x  and  x(t)  of  the  state  and  of  course 

o 

t.  In  the  area  of  LF  games,  the  type  of  strategy  spaces  U  and  V  which  were 

considered  and  treated  successfully  in  the  previous  literature  where  the  spaces 

of  piecewise  continuous  functions  of  time.  In  this  case,  the  problem  of 

deriving  necessary  conditions  for  the  Stackelberg  differential  game  with  fixed 

time  interval  and  initial  condition  x  ,  falls  within  the  area  of  classical 

o 

control.  Thus  variational  techniques  can  be  used  in  a  straightforward  manner. 

The  case  where  the  strategy  spaces  are  spaces  of  functions  whose  values  at 

instant  t  depend  on  the  current  state  x(t)  and  time  t,  i.e.,  u(t)  =  u|t  = 

u(x(t),t),  v(t)  =  V|t  =  v(x(t),t),  was  not  treated.  This  case  results  in  a 

nonstardard  control  problem  because  appears  in  the  follower's  necessary 

conditions.  Since  the  follower's  necessary  conditions  are  seen  as  state 

d  u 

differential  equations  by  the  leader,  the  presence  of  in  them  makes  the 


leader  face  a  nonstandard  control  problem.  In  this  Chapter,  the  nonstandard 
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control  problem  arising  from  the  consideration  of  the  above  strategy  spaces 
is  embedded  into  a  more  general  class  of  nonstandard  control  problems.  The 
characteristics  of  this  general  class  of  problems  are  the  following: 

(i)  each  of  the  components  u1,  of  the  control  m-vector  u,  depends  on  the 
current  time  t  and  on  a  given  function  of  the  current  state  and  time,  i.e. 
u1^  =  u^Ch^CxCt) ,t) ;  (ii)  the  state  equation  and  the  cost  functional  de¬ 
pend  on  the  first  order  partial  derivative  of  u  with  respect  to  the  state 
x.  The  vector  valued  functions  h1  may  represent  outputs  or  measurements 
available  to  the  i-th  "subcontroller,"  in  a  decentralized  control  setting. 

The  only  restriction  to  be  imposed  on  h1  is  to  be  twice  continuously 
differentiable  with  respect  to  x.  This  allows  for  a  quite  large  class  of 
hi,s  which  can  model  output  feedback  or  open  loop  control  laws.  It  can  also 
model  mixed  cases  of  open  loop  and  output  feedback  control  laws  where  during 
only  certain  intervals  of  time  an  output  is  available.  The  appearance  of 
the  partial  derivative  of  u  with  respect  to  x  prohibits  the  restriction  of 
the  admissible  controls  to  those  which  are  functions  of  time  only.  Two 
different  approaches  for  deriving  necessary  conditions  for  the  nonstandard 
control  problem  are  presented.  The  first  uses  variational  techniques,  while 
the  second  reduces  the  nonclassical  problem  to  a  classical  one.  The 
nonexistence  of  a  control  law  u*(x(t) ,t)  which  u*  solves  the  problem  for 
every  xq  is  shown.  The  nonuniqueness  of  the  solution  of  this  problem  is  also 
considered  and  explained.  The  results  obtained  for  this  nonstandard  control 
problem  are  used  to  study  an  LF  differential  game  where  the  players  have 
current  state  information  (x^ ,x(t) , t) ) .  Necessary  conditions  that  the  opti¬ 
mal  strategies  must  satisfy  are  derived.  The  inapplicability  of  dynamic 


13 


programming  to  LF  dynamic  games  is  explained  and  discussed.  The  singular 
character  of  the  leader’s  problem  is  proven  and  the  nonuniqueness  of  his 
strategies  is  proven  and  characterized.  In  particular,  it  is  shown  that 
commitment  of  the  leader  to  an  affine  in  the  current  state}time  varying 
strategy  does  not  induce  any  change  to  the  optimal  costs  and  trajectory. 

The  Linear  Quadratic  LF  game  is  also  worked  out  as  a  specific  application. 

We  will  now  outline  certain  generalizations  of  the  results  of  this  Chapter. 

We  consider,  first,  the  discrete  time  versions.  Consider  the  dynamic  system 

=  f(xk,u1(h1(xk,k)  ,k) , . . .  ,um(hm(xk,k)  ,k), 

ui<hl(Vk)  ,k^  » •  •  •  »u™(hm(xk,k)  ,k)  ,k) 

xq  given,  k  =  0,...,N-1 

and  the  cost 

N-i 

■J(u)  =  g^)  +  ^ZqLCx^u  (h  (xk,k),k) . um(hm(xk,k)  ,k) , 

u^(h1(xk,k)k) , . . . ,u“Chm(xk,k) ,k) . 

The  proof  of  the  corresponding  Theorem  3,2,1  is  straightforward.  An  immediate 
consequence  is  that  the  restriction 

ui(hi(xk,k) ,k)  =  A^hi(xk,k)  +  b£,  i=l,,,,,m1 

where  A^.B*  are  matrices,  does  not  induce  any  loss  of  generality  as  far  as 
the  optimal  cost  and  trajectory  are  concerned.  Proposition  3.2.1 
carries  over,  too.  A  discrete  time  version  of  the  LF  game  of  3-1  can  be 
defined  (see  [33]),  and  analyzed  as  in  3-3.  Several  information  patterns  can 
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be  exploited  by  employing  different  hX's  (see  [33]).  The  restriction  of  the 
leader  to  affine  strategies  can  also  be  imposed  in  the  discrete  case.  The 
case  where  higher  order  partial  derivatives  of  u  w,r.  to  x  appear  can  be 
treated,  and  all  the  analysis  carries  over.  This  case  is  of  interest  in 
hierarchical  systems  since  it  arises,  for  example,  in  an  N-level  LF  game  where 
the  players  use  control  values  dependent  on  the  current  state  and  time. 

Finally,  an  N-level  LF  game  where  on  each  i-level  (i=l, , , , ,N)^  n^  followers 
operate  (u*,.,,,u*  ),  play  Nash  (or  Pareto)  among  them,  and  u*|  = 

i  .  ni  .  ^ 

u. (h . (x, t) , t),  j=l , . . . ,n. ,  i=l, . . . ,N ,  with  given  h .  and  fixed  x  ,  t  ,  t ,  can 
JJ'  i  j  oof 

be  easily  treated  by  using  the  analysis  for  the  nonstandard  control  problem 
supplied  in  3-2. 

In  Chapter  4,  we  consider  a  continuous  time  two  player  deterministic 
differential  game  with  a  linear  state  equation  and  two  quadratic  cost  functionals. 
We  consider  the  case  where  the  players  have  at  each  instant  of  time  recall  of 


previous  values  of  the  trajectory,  i.e.  they  have  memory.  What  they  remember 
about  the  previous  values  of  the  trajectory,  is  allowed  to  change  with  the 

elapse  of  time.  It  is  a  known  fact  that  in  LF  and  Nash  differential  games, the  re¬ 
sulting  trajectory  and  strategy  values  vary  with  the  admissible  strategy  spaces. 

Most  of  the  results  available  until  now  deal  with  cases  where  the  current  state 
or  the  initial  state  of  both  of  them  are  the  only  available  information  to  the 
players.  A  more  general  situation  is  to  assume  that  at  each  instant  of  time, 
each  player  knows  something  about  the  previous  values  of  the  state  of  the 
system  and  about  the  previous  values  of  his  and  the  other  player's  decisions. 

The  first  attempt  to  derive  necessary  conditions  for  zero  sum  games  where  the 
strategies  depend  at  each  instant  of  time  t  on  the  part  of  the  state  trajectory 
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between  t-r  and  t,  where  r>0,  appears  to  be  in  [40 j,  In  f 41 J ,  [42J  the  zero- 
sum  case  is  considered  where  one  player  has  a  time  lag  information  on  the 

of  the  state*  In  [42]  a  Hamilton— Jacobi  theory  is  developed  for  such 
games.  In  Chapter  3  the  LF  differential  game  was  solved  when  the  leader's 
information  at  time  t  is  xCt) ,  x(0) ,  and  t.  It  was  shown  there,  that  the 
leader  can  in  general  restrict  himself  in  strategies  affine  in  x(t)  and  that  use  of 
nonlinear  strategies  inx(t)  will  not  improve  his  cost.  The  argument  of 
Chapter  3  can  in  principle  be  extended  to  the  case  where  the  leader's  infor¬ 
mation  at  time  t  is  (x(0),  t  <  0  <  t}  and  one  can  show  that  the  leader 

o 

does  not  in  general  deteriorate  his  cost  if  he  uses  strategies  affine  in 

{x(0) ,  t  <  9  <  t}.  Therefore  one  is  motivated  to  restrict  apriori  the 

t 

strategy  of  the  leader  to  be  of  the  form  £  [dgn(t ,s) ]x(s)  +  b(t),  in  which 

case  n  and  b  are  what  the  leader  will  actually  choose.  For  given  n  and  b 

the  follower  solves  his  problem.  Necessary  and  sufficient  conditions  for  the 

follower 's  control  problem  can  be  found  in  [45]  and  [46]  (Theorem  5.2)  respectively. 

On  the  other  hand,  the  leader's  problem  is  quite  difficult  since  his  unknowns 

are  n  and  b.  It  was  also  shown  in  Chapter  3  that  the  principle  of  optimality 

holds  in  LF  games  if  and  only  if  the  leader's  problem  can  be  treated  as  a 

team  problem  for  both  leader  and  follower.  (This  does  not  necessarily  mean 

^1=*'^2^ '  ^ese  remarks  motivate  us  to  study  LF  games  where  the  solutions  are 

linear  in  (x(9),  t  <  9  <  t}  and  constitute  a  team  solution  for  the  leader's 
o 

problem.  In  our  model  a  wide  range  of  delayed  information  structures  is 
included,  from  perfect  recall  of  the  previous  trajectory  to  recall  of  only  one 
previous  value  of  the  trajectory.  Cases  where  information  about  the  past 
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strategy  values  is  available  to  the  players  are  also  considered .  We  consider 
strategies  affine  in  the  available  information  and  represent  them  by  using 
Lebesgue-Stieltj es  integrals.  Both  Nash  and  LF  equilibrium  concepts  are 
considered  and  sufficient  conditions  are  developed  for  a  particular  but 
quite  interesting  class  of  problems.  Particular  emphasis  is  placed  on  the 
LF  case.  The  problem  that  we  deal  with  differs  from  those  considered  by 
Halanay  in  [40]  and  by  Ciletti  in  [41],  [42],  Halanay  considers  the  zero 
sum  case  only  and  he  allows  the  strategy  values  at  time  t  to  depend  on  the 
part  of  the  trajectory  between  t-x  and  t,  where  x>0  is  fixed.  Ciletti 
considers  also  the  zero  sum  case  and  allows  dependence  of  the  strategy 
values  at  time  t  only  on  x(.t  -a)  and  the  strategy  values  between  t-o  and  t, 
where  a>0  is  fixed.  Existence  and  uniqueness  conditions  related  to  the 
sufficiency  conditions  proved  in  this  Chapter  are  not  as  yet  known.  Our 
results  generalize  trivially  to  the  N  player  case  for  a  Nash  game  and  to 
the  one  leader-N  tollowers  case  for  an  LF  game.  Although  for  the  time  being 
our  results  are  not  accompanied  by  computationally  efficient  procedures, 
they  are  of  importance  sirce  they  provide  valid  characterizations. 

The  last  Chapter  5  deals  with  Nash  games  in  the  presence  of 
stochastic  disturbances.  It  is  known  that  the  study  of  stochastic  games  is 
usually  more  difficult  than  the  study  of  deterministic  ones  and  consequently 
very  few  interesting  and  general  results  are  known.  The  aim  of  this  chapter 
is  to  solve  some  special  classes  of  Nash  games  and  to  point  out  several  difficulties 
which  make  the  explicit  solution  of  such  games  very  difficult.  It  is 
shown  that  in  linear  quadratic  Nash  games^,  decoupling  of  the  information 
of  each  player  from  the  influence  of  the  control  of  the  other  is  not  always 
sufficient  to  establish  existence  and  uniqueness  of  solutions  affine  in  the 
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information.  Conditions  where  this  decoupling  property  sufficies  to  achive 
that  are  exhibited  for  both  static  and  dynamic  problems.  It  is  also  shown 
that  decoupling  of  the  information  of  each  player  from  the  control  of  the 
other  and  decoupling  the  cost  functional  of  the  one  player  only  from  the  control 
of  the  other  is  very  close  to  being  sufficient  for  existence  and  uniqueness  of  affine 
in  the  information  solutions  .  Conditions  where  this  is  the  case  are  exhibited. 

We  start  in  Section  5.2  by  considering  a  static  stochastic  Nash  game,  where 
each  player  has  a  quadratic  cost  and  his  information  is  a  linear  function  of 
a  Gaussian  random  variable.  Certain  known  results  are  considered  first  and 
some  new  ones  are  provided  concerning  the  existence  and  uniqueness  of  the 

solution.  In  5.3  we  study  solutions  of  the  game  of  5.2  which  are  affine  in  the 

information  and  provide  a  method  for  finding  them.  In  5.4  we  generalize  some 
of  the  results  of  the  previous  sections  to  the  case  where  each  player's  control 
vector  is  subdivided  to  smaller  control  vectors  each  one  of  which  has  to  use 
different  information.  The  information  available  to  the  subvector  of  each 
control  vector  is  nested.  In  5.5  we  consider  a  discrete  time  stochastic  Nash 
game  with  linear  stochastic  state  equation  and  quadratic  costs  where  the 
players  have  noise  corrupted  state  measurements.  Special  cases  of  this  game 
were  solved  in  [17]  and  [63].  The  case  where  both  players  have  perfect  state 
measurements,  was  studied  in  [17]  and  it  was  shown  that  if  the  noise  in  the 

state  equation  is  nondegenerate,  then  the  game  admits  a  unique  solution  affine 

in  the  information,  under  invertibility  conditions  for  certain  matrices.  It 
was  also  shown  in  [17]  that  if  the  noise  in  the  state  equation  is  degenerate, 
then  the  game  will  have  in  general  an  infinite  number  of  nonlinear  solutions. 
The  case  where  at  each  stage  k  the  players  share  their  previous  state 
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measurements  and  their  information  differ  only  in  the  k-th 
state  measurement  was  studied  in  [63]  where  it  was  shown  that  the 
game  will  admit  a  unique  solution  affine  in  the  information,  under 
invertibility  conditions  for  certain  matrices.  In  the  more  general  case 
where  the  assumptions  of  [17],  [63]  concerning  the  information  of  the  players 
do  not  hold, the  solutions  of  the  game  becomes  extremely  difficult.  In  Section 
5.5  we  single  out  some  new  classes  of  problems  which  can  be  relatively  easily 
solved  by  using  the  results  of  the  previous  Sections.  In  the  last  section 
We  translate  some  of  the  results  of  Section  5.5  to  a  continuous  time 
stochastic  Nash  game  with  a  linear  stochastic  state  equation  and  quadratic 
costs  where  the  players  noise  corrupted  state  measurements.  Examples 
demonstrating  certain  properties  of  the  solutions  are  considered  in  most 
Sections . 

Finally  we  have  a  Conclusions  Chapter  6  where  we  outline  directions 
for  future  research  and  seven  Appendices. 

1.3.  Notation  and  Abbreviations 

Rn;  n-dimensional  real  Euclidean  space  with  the  Euclidean  metric 
II  II  or  I  1 :  denotes  the  Euclidean  norm  for  vectors  and  the  sup  norm  for  matrices 
denotes  transposition  for  vectors  and  matrices. 

For  a  function  f:  Rn  ->■  Rm  we  say  that  f  €  C^  if  f  has  continuous 

mixed  partial  derivatives  of  order  k.  For  f:  Rn  -*■  R,  Vf  is  considered  as 

nxl  column  vector  and  f  denotes  the  Hessian  of  f.  For  f :  Rn  -»■  Rm,  Vf  is 

xx 

considered  as  n*m  matrix  (Jacobian).  For  f:  RnxR^*  Rm,  where  x€Rn,  y  €  Rk, 
we  denote  by  or  f  or  V  f  the  Jacobian  matrix  of  the  partial 
derivatives  of  f  with  respect  to  x  and  is  considered  as  n*m  matrix. 
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C([t0,tf],Rn)  = 


C  : 
n 


The  Banach  spaces  of  continuous  function  cp:  [tQ,tf] 
9;  [ tQ, tf ]  —  Rn ,  with  norm 

ii  9  I!  =  supl  l9(t)  I  ;  t  € [  tQ,  t f ]  } 


Ll([t0>tf3  >R  )  -Li>n: 


Loo(f  t0’  cf^  ,R  ^ 


NBV([tQJtf] ,Rn)  * NBV : 


where  I  I  denotes  the  usual  Euclidean  distance  in 

n 
R  . 

The  Banach  space  of  Lebesgue  integrable  functions 

cf 

9:  [t~,tJ~Rn  with  norm  11911s  1 9 ( t )  | d t . 

C0 

The  Banach  space  of  Lebesgue  measurable  functions 
which  are  almost  everywhere  bounded,  with  norm 
!|9li  *  ess  supL  i 9(  t)  I ,  t6[tQ,  tf]  j  . 

The  Banach  space  of  normalized  functions  of  bounded 
variation,  i.e.:  continuous  from  the  right  on  (t^.t^.), 
zero  at  tf,  and  !l9ll=Var(9)  for  96-NBV. 


A  norm  in  one  of  these  spaces  is  denoted  sometimes  by  ||*  jj  ,  I! 

k  Vfc  _  rk 

B  denotes  the  conjugate  space  of  a  Banach  space  B.  If  x  CB 
*  * 

write  <x  ,x>«  x  (x) . 
w.r.  to:  with  respect  to 
w. l.o.g.:  without  loss  of  generality 
n.b.d.:  neighborhood. 

Additional  notation  is  introduced  when  needed. 


'^Ibv 

and  x6b,  we 
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CHAPTER  2 

CLOSED  LOOP  NASH  STRATEGIES 


2.1.  Introduction 

The  present  Chapter  deals  with  closed  loop  Nash  solutions  to  continous 
time  differential  games  and  is  divided  into  three  parts,  2.2,  2.3,  and  2.4. 

In  2.2  and  2.3  we  are  concerned  with  existence  of  solutions  to  linear  quadratic 
Nash  differential  games  and  in  2 .4  with  uniqueness  of  analytic  solutions  to  Nash 
differential  games  with  analytic  data. 

In  2.2  we  consider  the  linear  quadratic,  time  invariant  case  over 
an  infinite  horizon.  The  structure  of  2.2  is  the  following.  In  2.2.1  we 
describe  the  system  and  formulate  the  problem  which  is  the  existence  of 
linear  closed  loop  Nash  solutions  which  result  in  a  stable  system.  The  ques¬ 
tions  posed  are  pursued  in  2.2.2  and  2.2.3.  The  existence  of  such  solutions 
depends  on  the  existence  of  solutions  to  a  system  of  two  coupled  algebraic 
Riccati  equations  which  result  in  a  stable  closed  loop  system.  Conditions 
for  the  existence  of  such  solutions  are  derived  via  Brouwer's  Fixed  Point 
Theorem.  The  conditions  derived  in  2.2.2  state  that  linear  closed  loop  Nash 
strategies  exist  if  the  open  loop  matrix  A  has  a  sufficient  degree  of  stability 
which  is  determined  in  terms  of  the  norms  of  the  weighting  matrices.  2.2.3 
contains  some  extensions  of  the  conditions  derived  in  2.2.2  which  do  not 
require  stability  of  the  open  loop  matrix. 

In  2 . 3  we  consider  the  linear  quadratic  finite  time  (t£[0,T])  case. 

The  matrices  involved  are  piecewise  continuous  functions  of  time.  The  struc¬ 
ture  of  2.3  is  the  following.  In  2.3.1  we  describe  the  system  and  pose  the 
problem  which  is  the  existence  of  linear  closed  loop  Nash  solutions.  The 
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existence  of  such  solutions  depends  on  the  existence  of  solutions  to  an 
associated  system  of  two  coupled  Riccati  differential  equations  over  [0,T]. 
In  2.3.2  sufficient  conditions  for  existence  are  derived  by  using  a  simple 
result  from  the  theory  of  differential  inequalities. 

In  2.4  we  consider  a  Nash  differential  game  where  the  functions  f, 
I’i’gi’  *-nv°lvec*  *-n  t^e  description  of  the  problem  are  analytic  functions  of 
their  arguments  and  seek  Nash  solutions  which  are  also  analytic  functions  of 
their  arguments.  This  problem  is  studied  in  2.4.1.  In  2.4.2  we  apply  the 
results  of  2.4.1  to  a  linear  quadratic,  finite  time,  Nash  game  where  the 
matrices  involved  are  analytic  functions  of  time. 

Four  Appendices  to  this  Chapter  are  given  at  the  end  of  the  Thesis. 


2.2.  Infinite  Horizon:  Existence  of  Closed  Loop  Nash  Strategies  and 
Solutions  to  Coupled  Algebraic  Riccati  Equations 

2.2.1.  Problem  Statement 

Consider  the  dynamic  system  described  by 

x  =  Ax  +  +  B2u2>  x(0)  =  xq,  t£  [0,-H»)  (2.1) 

and  two  functionals 

+oo 

J1(U1,U2')  =  |  (x,Q1x+  u^R11u1+  u2RL2u2)dt 

+00 

WV  =  l  (x'V+U2R22U2+UiR21Ui)dt 

‘“i  4i*2 

where  x,  u^,  u2  are  functions  of  time  taking  values  in  R  ,  R  ,  R  z  respec¬ 
tively  an.  A,  B^,  B2>  =  Q\,  R„  =  R^ ,  R^  >  0,  i,j  =  1,2  are  real  constant 

matrices  of  appropriate  dimensions. 

*  * 

The  problem  is  to  find  u^,  u2  as  linear  functions  of  x,  i.e., 


(2.2) 


m,  m„ 
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*  *  ~k  jc  k 

=  -L^x,  with  real  constant  matrix,  such  that  JjO^jU  )  is  finite  (see 


Appendix  B  ),  i  =■  1,2  and 


if  ^  ^ 

JL  =  J1(u1,u2)  £  J1(u1’u2>  f0r  every  U1  = 


if  y-  y^>  -u 

J2  =  J2(ul}u2)  s  J2^u1,U2^  f0r  every  u2  = 


(2.3) 


The  conditions  (2 . 3)  are  the  Nash  equilibrium  conditions.  It  is  known  (see 

k  k 

[1]),  that  a  necessary  condition  for  the  existence  of  such  controls  u^,  u^, 
is  that  there  exist  constant  real  symmetric  matrices  K^,  K2  satisfying 

0  ■  K1A  -  A'K1  +  %  -  KlBAkKl  -  K1B2R22B;K2  -  K2B2E22B2  K1 


+  k2B2R22R12R22B2K2 


0  =  K  A  +  A'K„  + 


2  +  Q2  -  -  K2BlRllBiKl  -  KlBlRUBiK2 


r }  b  '  k_  + 


(2-4) 


+  KiBiBnR2iRiiBiKi' 


It  can  be  proved  that  if  such  K/s  exist  and  the  closed  loop  matrix 


X  -  A  -  -  B24BiK2 


(2.5) 


has  Re\ (A)  <  0,  i.e.,  A  is  asymptotically  stable  (A.S.),  and 


+  K'.B.r"1^  .R^B’.K.  a  0,  i  *  j,  i,j  =  1,2 
J  J  J  J  J  J  J  J  J 


(2.6) 


For  (2.6)  to  hold  it  suffices  for  example:  s  0,  R..  s  0. 


then  the  strategies 
* 

u.  =  -L.x 
1  1 


B.'K.x, 
1  l  ’ 


i  =  1,2 


(2.7) 


•  )!*  ^ 

satisfy  (2.3)  and  J^,  are  finite.  (In  relation  to  this  see  Proposition  1 
in  [4]  .  The  proof  of  this  Proposition  in  [4]  does  not  hold  under  the 
assumptions  stated  there,  see  Appendix  A). 

In  the  next  section  we  will  deal  with  the  solution  of  (2.4)  and  try 


to  find  conditions  under  which  solutions  exist  and  yield  A  A.  S. 


2.2.2.  Conditions  for  Existence  of  Solutions 


We  start  by  introducing  the  following  notation 
A  0 


K  0 

K  =!  ,  ,  F  * 

0  Kj  0  A 


Q  = 


0  oi 

,  J  = 


<2. 


I  0 


Si  -  B,  R. .  B ' ,  S  = 


'1  "1“11"1.»  °2  ~  B2R22B2 
Sol  "  B2R22R12R22B2’  So2  =  ^^l^l*!!8! 


(2.8) 


s  -  Sl  °  ,  s  -  S*  0 

,°  S2  °  !°  So2 


where  I  denotes  the  nxn  unit  matrix.  Using  this  notation,  (2.4)  assumes  the 
form 


0  *  R(K)  “F’K+KF+Q-  KSK  -  KJSKJ  -  JKSJK  +  JKJS  JKJ. 

o 


(2.9) 


Consider  the  space  X  of  2n x  2n  real  symmetric  constant  matrices 
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M  0 

Y  = 

0  N 

'  j 

where  M  and  N  are  n x  n.  X  is  a  linear  subspace  of  the  space  of  2nx  2n  real 
matrices.  All  norms  of  the  matrices  to  be  considered  here  are  the  sup  norms 
(||A||  ■  sup{||Ax||  :jjx||  =  1}) ,  and  the  norms  of  the  vectors  are  the  square  root- 
Euclidean  norms.  It  is  easy  to  see  that  for  Y €  X,  ||y!|  =  max  (||m||,||n||)  .  We 
denote  by  IQ  the  2n  x 2n  unit  matrix  and  for  R  a  0  we  set  BR  =  [Y 6  X : 1 1 Yj j  £  R], 
i.e.,  B„  is  the  compact  ball  of  radius  R  centered  at  the  zero  element  of  X. 

We  define  the  function  §  from  X  into  X  by 

i(K)=ft(K)  +  K.  (2.10) 

Clearly  if  K€  X  then  5 (K) €  X,  and  i  is  continuous.  The  following  lemma  is 
proved  by  using  .Brouwer's  fixed  point  theorem  (see  [13]p.  161). 

Lemma  2.1.  If  for  some  R>0 

(3l|s||  +  1|SoU)R2  +  (||lo+2F|i  -  1)R  +  II oil  £  0  (2.11) 

holds  then  there  exists  X,  with  ||Kj|  £  R  which  satisfies  ft(K)  =  0. 

Proof.  For  X  a  fixed  real  number  we  have 

?(K)  =  K(F  +  XI  )  +  (F '  +  (l-X)I  )K  +  Q  -  KSK  -  KJSKJ  -  JKSJK  +  JKJS  JKJ 
0  0  o 

from  which  for  K£  Bousing  the  obvious  fact:  ||j||  =  1,  we  get 

l|*(K)||  ^  r(||xio+f||  +  ||(i-x)io+f|!)  +  ||q||  +  r2(3||sI|+  ||so!|). 

Since  ||lo+2F'!|  £  HX^+F1]  +  |j  (1-X)  IQ  +  Fll ,  with  equality  for  X  =  j  ,  we  set 
X  =  -j  ,  (best  X).  The  result  now  follows  by  direct  application  of  Brouwer's 


theorem. 
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Let  us  introduce  the  transformation 


K  =  a  K 


(2.12) 


where  or  t  0  is  a  constant  and  k€  X.  Substituting  K  =  in  (2.9)  we  obtain 

0  =  R(K)  =  R(«K)  =  (K)  -  («F)'K+  K(ofF)  +  Q  -  K(o2S)K  -  KJ(o2S)KJ 

-  2  -  -  2  -  (2.13) 

-  JK(o-  S)JK  +  JKJ(cr  S  )JKJ. 


Applying  Lemma  2.1  to  §  (K)  -  ft  (K)  +  K,  we  obtain  that  if  for  some  R  Ss  0  it 

a  a 


holds 


(3||s|1+!|So11)o2R2  +  (||I+  2«A!j  -  1)R  +  |'|Q|!  £  0, 


(2.14) 


then  there  exists  K£  X,  I!k]|  ^  R  which  satisfies  ft  (K.)  =  0.  But  then  K  =  aK 

1  Cl 

satisfies  ft (K)  =  RQ(K)  =  0  and  IJk'/,  s  |o?|r.  We  thus  have  proved 

T^mtna  7.2.  if  for  some  a  to,  R>0,  (2.14)  holds,  then  there  exists  K£X, 

I1k||  £  ! cy |  R  which  satisfies  r(K)  =  0. 

The  scaling  introduced  in  (2.12)  helps  to  improve  (2.11)  and  get 

(2.14),  because  in  proving  Lemma  2.2  we  applied  Lemma  2.1  to  a  whole  class  of 

§  '  s  which  are  nonlinear  (quadratic  in  K)  and  asked  that  at  least  one  of  them 
a 

have  a  fixed  point  via  Brouwer's  theorem.  As  it  turned  '"it  c  one  of  them, 
say  3—  has  a  fixed  point  then  all  of  them  have  (since  ft^(S)  =  ft(&S)  *  ftg(~S)) 
although  (2 . 14)  may  not  hold  for  $  t  a. 

Set 

a  =  3||S||  +  IJ.Sjl 

b  =  ||  1+  2aAj[ 

q  -  m 

e  *  V  qa  . 


(2.15) 
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Then,  (2.14)  assumes  Che  form 

aa2R2  +  (b-l)R  +  q  -  0  (2.16) 

If  a  =  0  then  =  0  and  B2  =  0,  and  the  game  is  meaningless  as  such.  There¬ 
fore  assume  a  f  0.  Inequality  (2 . 14)  is  satisfied  for  some  R  2  0  if  and  only 
if 

(i)  1  a  b  +  2|o,|  e 

or 

(ii)  q  =  J  and  1  <  b. 


In  case  (ii)  R  =  0  is  the  only  solution  to  (2.16)  and  thus  Lemma  2.2  guarantees 
only  the  solution  =0,  K2  =  0.  Consequently  we  will  concentrate  on 

case  (i),  i.e.,  when 

1  a  ||l  +  2a  Aj|  +  2ja|  e-  (2.17) 


If  (2 . 17)  holds  then  (2 . 16)  is  satisfied  for  all  R:  R^  5  R  s  ^  where 


*1,2 


=  l-b±,V(b-l)2-4a2e2  ^  0> 


„  2 
lot  a 


(2.18) 


In  this  case  Lemma  2.2  guarantees  the  existence  of  solutions  Kj.,  K2  with 
IIkJI,  ||K  II  *  |o|r.  we  have 

Theorem  2.1.  Let  a>0.  If  for  some  a  t  0,  (2 . 17)  is  satisfied,  then  for 
every  R :  R^  £  R  s  R2  where  R^,  R2  are  as  in  (2.18),  there  exist  K^,K2  satisfy¬ 
ing  (2.4)  such  that 


\ll 


s  of  R  s 


a 


R2£ 


2n-An. 


3!lS  'V'S 


M. 


(2.19) 


Proof.  The  proof  has  already  be~n  given  except  for  the  right  hand  side  of 
(2.11).  Since  1  -  ||I+  lot  A-  2  a  a!|  s  "l+  2a  A1  +  2  M'IaM  we  have 
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1  ■  bs  2 1  £y  1 1!  Aj] 


Notice  in  passing  that  M  is  independent  of  the  magnitude  of  a.  Before 
giving  the  next  Theorem  which  provides  us  with  necessary  and  sufficient 
conditions  for  the  existence  of  an  a  ^  0  satisfying  (2.17),  we  prove  the 
following  Lemma. 

L«-mna  2.3.  Let  T  be  a  real  nXn  matrix,  Y  and  P  real  numbers  Y  1 0, 
and  \(T)  =  ff  +  jw  be  any  eigenvalue  of  f •  Then 

(i)  If 

||l  +  Yr!|  5  P  (2.20) 

then 

«7+i)2  +  W2  S  (^)2.  (2-2« 

(ii)  if  y  >  0  and  p  *  1,  then  a  <  0  or  a  =  w  =  0  for  every  X(T). 
(iii)  If  y  <  0  and  P  =  1,  then  a  >  0  or  c=  w  =  0  for  every  X(T). 

(iv)  Let  T  =  T^T  \  where  A  is  the  Jordan  canonical  form  of  F . 

We  set 

p'  =  |lT'j'![T"1l|  Cp  ’  a  1). 

If 

!U+yaII  ^  ^ 

then  (2.20)  holds. 

(v)  If  A  is  diagonal  and 

12  2  P  2 

(^  +  7>  +  w^Crry) 


(2.22) 
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holds  then  (2.20)  holds.  In  particular,  if  f  is  symmetric  then  (2.28) 

2 

is  equivalent  to  (2.21) 

Proof.  (i)  Let  v  be  an  eigenvector  of  r  corresponding  to  &  +  jw  =  \(T) 
and  |]v|l  =  1,  w.l.o.g.  Then 

P  a  !!l  +  Yr||  a  ||  (I  +  YF)v||  =  |l  +  Y(cr+jw)| 
and  (2.21)  follows. 

(ii)  This  follows  trivially  from  (2.21)  by  noticing  the  (2.21) 
corresponds  to  a  disk  with  center  at  ~  and  radius  ,  which  in  case 

Y  >  0  and  P  s  1,  lies  in  the  left  half  plane  of  the  (o,jw)  plane. 

(iii)  See  (ii)  above. 

(iv)  Trivial. 

(v)  This  follows  by  using  (iii).  If  T  is  symmetric  then  T1  =  T_1 
and  ||t'!|  =  ||t'|=  V\max(T'T)  =  1  and  thus  p  1  =  1.  D 

Theorem  2.2.  Let  X(A)  =  a  +  jw  be  eigenvalue  of  A. 

(i)  If  e  =  0  and  (2.17)  is  satisfied  for  some  a  /  0  then  for 
X (A)  it  holds 


^Che  assumption  that  A  is  diagonal  in  (iv)  is  essential.  As  a  counterexample  let 

> 

-2  1 

r  =  A*  ,  T  =  T  =  I,  P  =  1 

0  -i 

\  I 

in  which  case  (2.21)  is  satisfied  for  all  Y:  0  <Y  £  4  but  for  x  =  — ~( 1,1)’, 

; - o —  "/  2 

!!x|!  =  l:!|(I  +  YDx"i  -,/l+Y  /4>  1. 


1 
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o  =  ReX (A)  <  0  or  X (A)  =  0  if  a  >  0 


a  ~  ReX (A)  >  0  or  (A)  =  0  if  ff  <  0 


.  1^2  .  2  ,  1.2 

«j  +  -)  +  „  s  (-) 


(2.23) 


l-llI+2aAll 


(2.24) 


(ii)  If  e  #  0  and  A  =  -si,  then  any  satisfies  (2.19)  and 


|k1!!,!|k2||  ^  «R2  •  J- 


(2.25) 


(iii)  If  e  f  0,  A  f  -si  and  (2.17)  is  satisfied  for  some  a  f  0,  then 


for  any  X (A)  it  holds 


o  <  -e  or  X (A)  =  -e  if  a  >  0  (2.26) 

u  >  +c  «  X  (A)  =  e  if  or  >  0 

and  HkJI  ,  ||K21L  satisfy  (2.11).  Moreover  |  cy  J  • 

(iv)  If  A  ¥  -si  (A  t  el)  is  diagonalizable,  A=TAT  ^  where  A  is  the 
Jordan  canonical  form  of  A,  p  1  =  11  tH  - H t  j]  and  for  some  y  >  0  (y  <  0)  it  holds 


(ct+ e  +  ^-)2  +  w2  s  — ~  ,((a  -e+7>2  +  w2  ^  — ^r) 

Y  p’Y2  Y  p’Y2 


(2.27) 


then  2(1+ gy)  ^  2(1  -ey)^  satisfies  (2.17) 

Proof.  (i)  (2.23)  follows  from  Lemma  2.3  (i,ii,iv)  and  (2.14)  from  (2.19). 

(ii)  The  first  part  is  trivial.  (2.25)  follows  as  (2,24). 

(iii)  Let  b  =  |~|  .  Then  (2.17)  yields 
b  -  e  s  Hbl  +  All  if  o:  >  0 

b  -  s  s  'lbl+  A|1  if  a  ^  0. 


J 
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Necessarily  b  -  e  >  G.  Let  y  =  "(b-eT 

1  a  Hl+Y(A+  el)]!  if  a  >  0 
1  s  I'l  +  Y  (-A+  el)1!  if  of  <  0. 


then 


We  set  F  =  +  A  +  el  and  we  have 

1  a  !U  +  Yr!|.  (2.28) 

(2.26)  follows  now  from  Lemma  2.3  (ii)-(iii). 

(iv)  We  bring  (2.17)  to  the  form  (2.28)  and  apply  part 

(iv)  of  Lemma  2.3.  D 

If  A  is  symmetric,  then  the  existence  of  y  satisfying  (2.27)  is 
equivalent  to  the  existence  of  a  satisfying  (2.17).  For  (2.17)  to  hold  it 
suffices  1  a  Art  (I+2orA) 1  (I+2oA)]  +  2|or|e.  By  using  the  fact  !|m||2  s  tr(M’M) 
it  follows  that  the  existence  of  an  a  >  0  satisfying  (2.17)  is  guaranteed  in 
the  following  two  cases  (assuming  A  is  not  a  scalar) . 

(i)  trA  £  -e 
trA'A  <  e2 

A  =  (n-l)trA'A  +  (trA)2  +  2strA  +  (2-n)e2  a  0 

JL  j.  -  (6+trA)  zJL 
2e  2(e2-trA'A) 

(ii)  trA'A  >  e2 
trA  s  -e  • 

We  shall  now  consider  the  stability  of  the  closed  loop  matrix  A 


(2.5).  Let 
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a'  -  4l!sH  +  ilsjl 

(e'2  -  s2  + »si|-N!) 

1  2~  2  2 
1-b  +y  ( l~b)  ~4 q;  e 


l-b+^/a-b)2-4a2£,2 

2a  2a' 


where  a,  e,  q,  b  are  as  in  (2.15). 

Theorem  2.3.  Let  Re[X(A)]  <  0 

(i)  If  for  some  a  >  0  it  holds 

1  a  !jl+  2o?A]|  +  2a  e 

1  >  ||  I  +  2«a|| 

a2||S!lR2  <  a2Hsol|R2  +  !!  <j| 

then  there  exist  ||K.||  S  aR^  i  =  1,2,  solving  (2.9)  and  A 

A.S. 

(ii)  If  for  some  <»  >  0  it  holds 
1  a  ||l  +  2aAl|  +  2ae  ' 

1  >  l|l+  2aA|| 

IIq’,1  or  HsJI  i  0 

then  there  exist  K^,  K2,  j ! K_^ | ]  £  i  =  1,2,  solving  (2.9)  and  A  (2 

A.S.  . 

Proof,  (i).  (2.30.1)  makes  Theorem  2.1  applicable.  We  have 

lil+2aA!i  =  "I  +  2a(A-S1aK1-S2a'K2)!1  £  !!l+2*Al!  +  %2,;S!||!Kl! . 


(2.29) 


(2.30.1) 

(2.30.2) 

(2.30.3) 

:.5)  is 

(2.31.1) 

(2.31.2) 

(2.31.3) 

.5)  is 
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Since  by  Lemma  2.3  A  will  be  A.S.  if  II  I  +  2all  <  1  for  some  a  >  0,  it  suffices 
1!I+  2a A] j  +  4q'2'!s'|'!k1|  <1. 

(2.30.2)  implies  >  0.  Using  K  =  aK  and  (2.19)  we  have  that  it  suffices 

('  1+  2aAl|  -  1)R2  +  4a2||s|lR2  <  0 

and  since  (2.14)  holds  for  R  =  R2  it  suffices 

4a2|ls!|R2  <  OI'S'1  +  ||So||)a2R2  +  |!qH 

which  is  equivalent  to  (2.31.3) 

(ii):  (2.31.3)  implies  that  the  inequality 

a’a2R2  +  (S!l+2aA||  -  1)R  +  q  s  0 

is  satisfied  for  all  R:  R’  S  Ri  R1  where 

1  2 


R2  £  R|  a  0.  (2..31.2)  implies  R2  >  0.  If  the  above  inequality  holds  for 
some  R  and  a,  then  (2.14)  holds  for  the  same  R  and  a  since  a  £  a1.  Therefore 
there  exist  K^,  K2>  IkJ]  £  aR^,  i  =  1,2  solving  (2.9).  Repeating  the 
analysis  of  (i)  we  have  that  for  A  to  be  A.S.  it  suffices 

4a2Hsj|R^2  <  (4iiSl|  +  j|So'j)a2R’2  +  ||q!| 

which  holds  by  (2.31.3).  u 

If  equality  is  allowed  in  (2.30.3)  or  |jQ||  =  ;|SQ  =  0  in  (2.31.3) 
then  the  conclusions  of  (i)  and  (ii)  in  the  given  Theorem  change  and  allow 
A  stable,  i.e.,  Re  [ X (A ) ]  -  0. 
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The  geometric  interpretation  of  the  conditions  given  in  Theorems  2.2 
and  2.3  is  given  in  Figs.  1  and  2.  Figure  1  corresponds  to  Theorem  2.2,  parts  i, 
ii,  iii,  which  say  that  a  necessary  condition  for  the  existence  of  an  a>0  satis¬ 
fying  (2.17)  is  that  the  eigenvalues  of  A  lie  in  a  disk  centered  at  with  radius 
r  =  ~  -e  ,  for  some  a  >  0,  which  is  equivalent  to  saying  that  all  \(A)'s  lie 
in  the  open  half  plane  on  the  left  of  the  line  e^,  (a=-e),  or  at  -e.  Figure 

2  corresponds  to  Theorem  2.2,  part  (iv),  where  it  is  assumed  that  A  is  diagonaliz- 

able .  It  shows  that  if  the  eigenvalues  of  A  lie  in  a  disk  as  in  Figure  2  with 

radius  r  =  — —  -  -  e )  and  centered  at  then  this  a  satisfies 

/ — r  lot  lot 

(2.17)  and  thus  (2.9)  has  a  solution.  If  A  is  symmetric  then  p'  =  1  and 
0  =  90°.  If  a  <  0,  then  we  have  the  mirror  images  with  respect  to  the  jw 
axis  of  the  circles,  cones,  lines  depicted  in  Figures  1,  2. 

Employment  of  a  different  function  $  in  (2.10)  and  application  of 
Brouwer's  theorem  may  in  general  provide  different,  perhaps  better,  existence 
results.  Another  suitable  $  can  be  defined  as  follows.  If  K  solves  (2.9)  and  A 

(and  thus  F)  is  A.S.,  then  K  =  —  ,  where  a>0,  solves  equivalently  (see  Appendix  C) 

£=  J^°eaFt[Q-toSK-KJa2SKJ- Jfe2SJK+JKJa2SoJKJ];ea'F  Cdt.  (2.32) 

Let  (K)  denote  the  right  hand  side  of  (2.32).  Let  also  A  =  TAT  ,  A 
being  the  Jordan  canonical  form  of  A  with  mxm  the  dimension  of  the  largest 
Jordan  block  (m  £  n) .  Let  also  a?  s  Le  as  in  (2.15)  »  ot>0,  and 

p  =  WiT"1!! 

\  =  max  Re  [X(A)]  <0  (2.33) 

m-1  .  ..  i+j+1 

tt(w)  =  I  (7)  for  w  >  0, 

i,j=0  L,J- 

TT  =  TT  ( 1/  (-0 !\)) 
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jw 


Figure  1.  Regions  of  eigenvalues  of  A  in  accordance  to 
Theorem  3.2  (i)-(iii). 


1 
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and  e(m>a)  >  0  be  such  that 

TT(l/e(m»a))  =  • 

Clearly  e(m,cO  exists  and  is  unique,  for  given  o?  and  m. 

Theorem  2.4.  Let  A  be  A.S.  and  T,  A,  m,  ir ,  e(m,ot)  as  above.  If  it 

holds  that 

-  ,  x  1 

s  -e(m,af)  •  -  (2.34) 

then  there  is  Kg  X  which  satisfies  (2.9),  and 

i|K||  *  a'^  =  a  — “  4qaa^  }  ^  .  (2.35) 

2aa  p  *tt  p4rr  ap 

2 

In  addition,  if  A  is  diagonalizable  (i.e.,  m  =»  1)  then  e(l,ar)  =  asp  for 
every  a  >  0. 

Proof .  Let  K  =  aK,  a>0,  IlKlI  <  R,R  ^  0.  In  order  to  use  Brouwer's  theorem 
we  ask  for  ||?a(K)i;  £  R  for  some  a  and  R.  It  suffices 

*i"oo  —  I 

||s  (K)H  =  ||  Jo  e^tQ- Ka2SK- KJa2SKJ  -  JKa2SJK+ JKJcySoJKJ] -eaF  tdt|| 

*  t||Q||+  (3!ls|l  +  ||So|l)a2R2]  Jo  l|eaFt||2dt  £  R 
or  by  using  (D-l)  (Appendix  D) 

ac?2R2  -  R  +  q  s  0  (2.36) 

P  TT 

which  holds  because  of  (2.33),  (2.34).  The  rest  is  easy.  D 

It  is  remarked  that  if  A  is  diagonalizable,  then  (2.34)  gives 
X  s  -ep  )  introduction  of  a  >  0  induces  no  improvement  of  the  result,  which 
is  in  agreement  with  the  fact  that  scaling  cannot  facilitate  the  existence  of 
solutions  of  (2.9).  Incase  A  has  Re  [\(A)J  >  0,  we  can  have  results  similar 
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to  those  of  Theorem  2.4  by  employing  or  <  0. 

The  geometric  interpretation  of  Theorem  2.4  is  given  in  Figure  3 

and  shows  simply  that  if  the  eigenvalues  of  A  lie  on  the  closed  half  plane 

£  Cm  &  ) 

on  the  left  of  the  line  e2>  (cr  - - s  ^  O ,  for  some  a  >  0  then  there  exists 

K  which  solves  (2.9).  If  A  is  diagonalizable  then  -  e ^  =  -eP  and  since 

pal,  the  line  e  is  on  the  left  of  In  this  case  a  combination  of 

Theorems  2 . 2  (iv)  and  2.4  gives  an  easily  verified  sufficient  condition  for 
solvability  of  (2.9). 

Finally  Theorem  2.3(ii)  can  be  interpreted  along  the  same  lines  as 
Theorem  2.2,  using  Figures  1  and  2,  where  e'  is  used  instead  of  e.  So, 
if  s'  ¥  0,  A  is  diagonalizable  and  all  \(A)'s  lie  in  a  disk  as  in  Figure  2 
with  e'  in  place  of  e,  then  (2.9)  has  a  solution  and  the  closed  loop  matrix 
A  is  A.S.  If  s'  -  0  and  A  is  diagonalizable  (then  j|Qjj  =  0,  e  “  0) ,  then  if 
the  eigenvalues  of  A  lie  in  the  interior  of  the  disk  in  Figure  2  the  same 
conclusion  holds.  The  version  of  Theorem  2. 3  with  <  0  and  Re\  (A)  >  0  results 
to  A  unstable,  i.e.,  Re  [\(A)]  >  0  and  is  thus  of  no  interest  to  us. 


We  close  this  section  with  five  remarks. 

(i)  The  only  assumption  on  Q± ,  R_  '  s  used  in  developing  the  proofs  in  this 
section  was  that  R^,  R22  exist.  Neither  Q.  s  0  or  «  0  or  R„  s  0,  nor  any 
controllability,  observability,  or  optimality  conditions  were  used. 

(ii)  If  Q  =  Q  +  KSK  +  JKJSqJKJ  at  0  (it  suffices  R12  and  R21  at  0)  and  A  is  A.S. , 


since 


“ !  ' 

°i 

_  !  K  -  -Q 
A  ' 


^  I 

a  standard  result  in  Lyapunov  theory  yields  K  a  0.  Since  =  x^lvx^^  (see 
[  4  ]  )  we  will  have  J*  a  0  for  every  xq,  as  it  should  be  expected  in  case 

Qi’  Rij  a  °’  i,j  =  1’2" 


Figure  3-  ^8ions  °f  eigenvalues  of  A  in  accordance  to 
Theorem  3.4. 
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(iii)  Consider  the  single  Riccati 

KA  +  A'K  +  Q  -  KBR* 1 B 1 K  =  0  (2.37) 

where  R>  0  and  A  is  A.S.,  with  Q  not  necessarily  positive  definite.  Then 
Brouwer's  Theorem  provides  results  which  can  be  used  to  easily  verify  whether 
the  frequency  condition  of  Lemma  5  in [11]  holds.  It  is  easy  to  prove  (as 
in  Theorems  2.1,  2.2)  that 

(i)  if  there  is  a  >  0  such  that 

1  a  jll+  2o'A|!  +  2a  ^|Q||||BR“1B,U  (2.38) 

then  (2.37)  has  a  solution  K.  If  in  addition 


a2||s!|R2  <  !|Qi| 

1  >  l|l+  2aA|| 

where 

l-lll+2aAl|  +V(l-llI+2aA!l)2-4llQi!llBR"1B'|l 

*2  Tpl 

then  A  -  BR'Vk.  is  A.S. 

(ii)  If  there  is  an  a  >  0  such  that 


(2.39) 


(2.40) 


1  a  ||  1+  2o?A||  +  2*/2  •  a  v7||q||||BR_1B' ||  (2.41) 

then  (2.37)  has  a  solution  K  and  A  -  BR  ^B'K  is  A.S. 

Let  now  Q  =  -C'C  £  0,  and  assume  (A,C)  observable,  (A,B)  controllable. 
Using  Lemma  5  of  [11],  we  see  that  (2.38)  and  (2.39),  or  (2.41)  imply  that 

I-  B'(-jw- A')‘1C'C(jw- A)_1B  =  0  for  all  real  w.  (2.42) 

*t~CO  I 

Also,  since  K  “  Jq  e^t(Q-  KSK)e^  ^  and  Q  £  0,  it  will  be  K  £  0. 

(iv)  In  order  to  guarantee  the  fixed  point  property  of  ?(K),  one  could  have 
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employed  a  contraction  mapping  machinery.  Then  §  should  map  a  closed  set 

Dq  C  X  into  itself  and  in  addition  ?  should  be  Lipschitzian  with  Lipschitz 

constant  L,0  5  L  <  1  on  Dq.  But  since  §  is  quadratic  in  K  Dq  should  be 

bounded  in  order  to  guarantee  that  I  is  Lipschitzian  there.  However  this 

amounts  to  Dq  compact  and  we  could  consider  Dq  a  ball  BRw.l.o.g..  So,  in 

order  to  use  the  contraction  mapping  Theorem  we  should  have  made  assumptions 

to  guarantee  L  <  1  in  addition  to  those  made  to  allow  the  use  of  Brouwer's 

Theorem  and  this  would  result  in  a  weaker  conclusion. 

(iv)  The  assumptions  of  Theorem  2.1  guarantee  the  existence  of  K^,  K2 

solving  (2.4),  which  lie  in  B  .  Thus  if  the  solution  of  (2.4)  is  unique  it  wil] 

1 

be  in  BR  .  If  not,  then  there  may  be  additional  solutions  K  K0  in  B„, 

1  i  2  R 

R>R.  which  are  not  in  BD  which  solve  (2.4). 

R1 

2.2.3.  Extensions 


I^t  us  now  try  to  relax  the  assumption  on  A  to  be  A.S.  Two  approaches 
will  be  considered.  In  both  of  them,  we  use  the  solution  of  an  appropriately 
defined  auxiliary  problem  in  order  to  show  existence  of  solutions  to  our 
main  problem  via  Brouwer's  theorem. 

Consider  first  the  optimal  control  problem 


i  *  Ax  +  [B  ‘B  ]  !  1  .  x(0>  -  x  , 

!  “2  , 

^  * 

min  Jo(x'Qx+  [u^Iu^JR  )dt 


t  €  [0,+®) 


(2.43) 


where 
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Rll+R21 


7S  Ql+Q2  ~ 

3  *  ~y~  >  R 


R22+R12 


*1 

0 


L 


o 

w 


(2.44) 


~  ~  3 

with  R^  ^  >  0.  Under  certain  assumptions  (controllability-observability 
and  Q>  0,  or  see  Theorem  2,  page  167  in  [14]  or  Remark  (iii)  in  (2.3)) 
there  exists  K  satisfying 


0  =  KA  +  A’K  +  Q  -  K(S  +i,)K 


with 


such  that 


§1  = 

B1R1  Bl’  ^2 

ui! 

1 

["r^BJKx 

I 

i 

s 

- 

f 

1 

U2  | 

2R2  B2 


solves  (2.43),  and  such  that 


(2.45) 


(2.46) 


A  =  A  -  (S1+S2)K 


(2.47) 


i.e.,  the  closed  loop  matrix  for  (2.-+3),  is  A.S.  Let 


=  K  +  Al,  K2  =  K  +  A2 


be  substituted  in  (2.4)  and  by  using  (2.45)  we  obtain 


(2.48) 


For  this  to  hold  it  suffices  R. . 

li 


>  0 


Ri j  £  ^  *- > 2* 
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0  =  A  A.  +  A.’ +  A1(S]_+ S2  -  Sx  -  S2)K  +  K(S1+ S2  -  Sj.  -  S2>A1 

+  42<Sol‘S2)R+  ^<Sol'S2)42  -  41S141  '  4 1S 24 2  '  42S241 

+  42Sol42+  2  +5(ll  +  S2+Sol-Sl-S2'S2^ 

0  -  A2A  +  A'A2  +  A2(S1+  S2  -  SL  -  S2)K  +  K(Sl  +  S2  -  Sj_  -  s2)A2 

+  Ai(so2-si)k+  k(So2-si)Ai  -  a2s2a2  -  a2s1a1  -  a^ 

+  Also2Al  +  ^  +  K(St+  S2+  so2-  st  -  S2  -  spic. 

where  S.j  S  .  are  given  in  (2.8). 
i  oi 

Let  a  be  as  in  (2.15)  and 

A.h  »1 

I  0  * 

-  ; 

v>  =  !!k(s1  +  s2-s1-s2)||  +H1 

"  m^{!|K(Sol- S2)!lJ!!K(So2- S^ll}  (2.50) 

Qi-Qp  -  ~  ~ 

q  -  nax{||-Y-£+K(S1+S2+Sol-S1-S2-S2)K|J, 

Q2'Q1 

II — —+  usl  +  s2+  so2  -  Sl  -  s2  -  SL)Klj} 


b  =  ||  1+  20TA||  +  2Q> 

~  m  l-b+J (l-b)2-4<^2a2q 

f\ 


The  proof  of  the  following  Theorem  is  similar  to  the  proofs  of 


Theorems  2.1  and  2.3. 
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-Theorem  2.5. ;  Let  A,  M ,  q  be  as  in  (2.47),  (2. 50,  a  +  0,  then 
(i)  If  for  some  a  >  0 


1  Ss  |  I"*"  2q/A| |  +  2of  (jj,  +v/a^') 


(2.51) 


then  there  are  A2  such  that  =  K  +  A^»  K2  =  K  +  A2  so^ve  ^-4)  and 

W^iW  s  “V 

(ii)  If  in  addition  to  (2.51) 


1  >  b 

o,2j| S||  R2  <  «2|tSo!!R2  +  2^lR2  +  q 


(2.52) 


then  the  closed  loop  matrix  A  (as  in  (2.51))  is  A.S. 


(iii)  If  for  some  a  >  0 


1  2  !|l+  2<*a!1  +  2a?  (^  +y  (a+ljs|i)q  "j 


(2.53) 


1  >  b 


q  or  !|sj|  or  ^  7s  0 

then  both  (i)  and  (ii)  above  hold. 

Proof.  The  equation  (2.49)  can  be  written 


0  =  A?  +  F'A  +  t( A) 


Qi-Q,  ~  ~  - 

_ _ _ h-JL  V  ( C 


r^-+K(S1+S2+Sol-S1-S2-S2)K 


VQi - 

-T-+K(S1+S2+So2-S1-S2-S1)K 


whe  re 


A  0 


0  A 
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Using  similar  methods  as  in  the  proof  of  Lemma  2.2, 
suffices  to  hold 

an2R  +  (II 1+  2o'Aj|  -  1  +  2 yP]  R  +  q  £  0 


we  conclude  that  it 


(2.54) 


in  order  for  (2.49)to  have  a  solution  where  ||  <,  q-r.  The 

rest  follows  as  in  the  proofs  of  Theorems  2.1  and  2.3.  a 

The  usefulness  of  the  presented  approach  is  clear  in  case  the 
game  is  used  to  describe  a  situation  where  two  independent  controllers 
desire  to  achieve  the  same  objective  using  slightly  different  information 
(Q^)  or  control  effort  (R  ) . 

Consider  now  the  two  independent  control  problems 


and 


x  =  Ax  +  BjUj.,  x(0)  =  xq,  t€  [0,-H») 

CO 

minimize  Jo(x'QiX  +  u{Rnu1)dt 

x  =  Ax  +  B2u2>  x(0)  =  xQ,  t  £  [0,-hu) 

00 

minimize  J^x'c^x  +  u2R22U2^dt* 


(2.55) 


(2.56) 


Under  proper  assumptions  the  two  Riccati  equations 


0  =  A'Kj^  +  KjA  +  Ql  -  KjS^ 


o  =  a'k2  +  k2a  +  q2  -  k2s2k2 


(2.57) 


have  solutions  K^,  K2>  the  =  -R^Sl^x,  u2  =  _R22B2^2X  solve  (2.55)  and 
(2.56),  and 


Aj_  "  A  A2  *  A  ^2^2 


(2.58) 


are  A. 5.  Let 
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4l  lj.r-K.r-  0 

1°  ‘2j  i°  V 

k  > 

K1  =  +  Ap  K„  =  K  +  A 


(2.59) 


A  = 


\  0 


Then  (2.5)  can  be  written 


0  -  F'A  +  AF  -  ASA  -  AJSAJ  -  JASJA  +  JAJSQJLJ  -  KJSKJ  -  JKSJK 


(2.60) 


+  JKJS  JKJ  -  KJSAJ  -  AJSKJ  -  JKSJA  -  JASJK  +  JAJS  JKJ  +  JKJS  JAJ. 

o  o 


Let  also 


n  -  |!ks|J  +  j|KJSoH  +  !|kjs!| 
q  -  IjKJSKJ  +  JKSJK  -  JKJS  JKJ||. 
b  =  ||  I  +  2ofF||  +  2 ow 


(2.61) 


R2  = 


b+J  (L-b)2-4o'2aa 

2o-2a 


Theorem  2.6.  Let  F,  A,  q  be  as  in  (2.59),  (2.61),  a  ^  0 
(i)  If  for  some  ar  >  0  it  holds 

1  2  ||I+  2orF||  +  2Q'[y,+A/a^  ]  (2.62) 

then  there  exist  A^  A2  such  that  Kj_  +  A^  K2  +  A2  solve  (2.4)  and  HaJIsaR 
(ii)  If  in  addition  to  (2.62) 

1  >  b 

2|!„iI“2 


Q'2,|s!!r2  <  a2ils  Mr2  +  2or(''KJS  '1  +  HKJS"  )R  +  q 
z  o  z  o  '  2  M 


(2.63) 
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then  the  closed  loop  matrix  A  (as  in  (2.5))  is  A.S. 

(iii)lf  for  some  a  >  0 

1  a  ||l+2»Fl|  +  2a[jI+va(qHJS||)]  (2.64) 

1  >  b 

q  or  '|  S  !|  or  jlKJSj  +  ||KJSj|  1  0 

then  both  the  conclusions  of  (i)  and  (ii)  hold. 

Proof .  Working  as  in  Theorem  2.5,  it  suffices  to  hold 

aa2R2+  [||I+  2ofF||  -  1  +  2aJT]R  +  q  £  0  (2.65) 

in  order  for  (2.60)  to  have  a  solution  A,  where  1Ia]|  £  crR,  and  so  on.  D 

The  usefulness  of  this  approach  lies  in  the  fact  that  the  results 
which  pertain  to  the  case  where  the  system  is  controlled  separately  by  the 
decision  makers  can  be  used  to  check  Luc  existence  of  the  solution  when  the 
two  decision  makers  control  it  jointly  and  use  Nash  strategies. 

The  Theorems  2, 2.-, 2. 3,2.4  and  the  interpretations  in  Figures  1,  2,  3 
hold  also  for  the  two  approaches  presented,  with  the  appropriate  modifica¬ 
tions.  For  example  for  Theorems  2. 5, 2. 6,  Figures  1,  2  and  the  first  approach 
one  should  use  A,  jj,  +  „/aq,  jj,  +  */(ad-|(SI()q  instead  of  A,  e,  e’  respectively. 

Finally,  note  that  the  existence  results  in  all  cases  developed 
previously  are  dependent  on  the  parameter  e  (or  e1).  Since  e  is  a  function 
of  the  weighting  matrices  and  since  rescaling  of  the  criteria  will  affect  the 
weighting  matrices  it  is  of  interest  to  point  out  how  this  scaling  affects 
the  existence  results.  Nothing  changes  in  the  game  if  we  have  J1  =  r  J 

ill 

instead  of  J^,  r^  >  0,  i  =  1,2.  So  considering  instead  of  Q.  , 
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R.  .  we  have 
ij 


'  Sol''  l!So2'il 


«  =  max(r.!;  Q-Jj  ,r  |j  Q  !l)  [3max(-— — ■  ,  -f- )  +  max(^ ,  )  ] 

i  i  z  z  rl  r2  1  r2 


e  =  eZ(r)  =  max( — —  ,  ||  Q2||)  •  [SmaxCrjjSjJI  ,j|S2|| )  +  max(r!|Soll! , !ISq2|!  )] 

r2  2 
where  r  =  —  •  Carrying  out  the  minimization  of  e  (r)  with  respect  to  r 

ri 

we  find  the  minimum  e 


e  -  J Jmax I 


""I 


1 » i !  Qo !  *  * 1  i  s  0 ! !  3  +  max  [ 


l!  ol  *  y2'  1  o2 


(2.66) 


The  r  at  which  e (r)  becomes  minimum  is  given  by  the  following.  Let 

l|s2ll  llso2!l  IISjII  l|s  2||  _  11^11 

r«  ’  ““'pgrp^ir’  •  r3  *  m*x(,Tsjf  ■ 

•  7^  __ 

If  r  s  r  Sr  then  r  is  any  point  in  [r,r  ]. 
a  3  or 

—  *  — 

If  ra  s  r  s  rp  then  r  *  r. 

■■  ^  n_ 

If  r  s  r_  s  r  then  r  is  any  point  in  [r  , r] . 

oc  3  P 

* 

For  e  as  in (2. 29)  the  same  analysis  holds  and  the  optimum  e 
is  given  by  a  relation  exactly  the  same  as (2.66)  but  with  4  multiplying 

the  first  term  instead  of  3-  We  can  consider  in  all  of  our  conditions  that 
e,  (e1)  is  given  by  (2.66).  Notice  also  that  a  similar  procedure  will  give 

the  minimum  values  of  p,  +  y/aq,  see  (2.50),  (2.61).  It  is  interesting  to 
notice  that  if^ra  then  all  r:  r  s  r  S  ra  give  the  same  e\ 

Actually,  as  (2.3)  indicates,  the  existence  of  solutions  for  the  game  should 
not  depend  on  multiplying  Jj  or  J2  by  a  positive  constant.  Our  conditions 
have  at  least  preserved  this  property  for  an  interval  [r,r^]  or  frg’r^' 
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2.3.  Finite  Horizon:  Existence  of  Closed  Loop  Nash  Strategies  and  Solutions 
to  Coupled  Differential  Riccati  Equations 

2.3.1.  Problem  Statement 


Let  us  consider  the  dynamic  system 


x  =  Ax  +  +  B2u2,  x(  0  )  =  Xq>  t  £  [0,  T] 

the  two  cost  functionals 


(2.67) 


Ji(VU2)=2  k(T)  'K.x(T)  +J  (x'Q.x+u^Riiui  +ujR_u^)dt] 

(2.68) 

i,  j  = 1,2,  i +  j 


and  the  associated  Nash  game,  see  [l].  The  state  x  and  the  strategies  u^,  u^ 

ml  m2 

take  values  in  R  ,  R  and  R  respectively.  The  matrices  A,  R^ ^ , 

are  real  valued  piecewise  continuous  functions  of  time  and  of  appropriate 
dimensions.  We  also  assume  K.  *  K.'  >0  constant  real  matrices,  Q,  (t)  =  Q.  (t)  '  >0, 

i  i  —  ii 

R  (t)  =  R„  (t)  '  >0,  Rii(t)  >0,  Vt€[0,T],  where  the  time  interval  [0,T]  is 
assumed  fixed. 

We  restrict  the  admissible  strategies  to  those  which  are  linear  in  x. 


i.e ; 


ui(t)  =  Li(t)x(t),  i  =  1,2. 


It  can  be  shown  ([1])  that  if  such  an  equilibrium  Nash  pair  of  strategies 
exist,  it  will  be  given  by 


u. 

i 


R .  f  B P . x ,  i  =  1,2, 

li  i  i 


(2.69) 


where  P^,  P2  satisfy  a  system  of  two  coupled  differential  Riccati  equations. 
This  system  can  be  written  as : 
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-P  =  F'P  +  PF+Q  -  PSP-  PJSPJ  -  JPSJP+JPJS  JPJ 

o 


(2.70) 


where 


P(T)  =  Ko,  t€[0,  T] 


F  = 


0  Ai 


S  = 


fBlRUBi 

i 

!  o 


B2R22B2 


B2R22R12R22B2 


Q  - 


i  0 

s 


o  : 


‘2/ 


BlRUR21RnB; 


) 


(2.71) 


J  = 


I  ' 


,  I  =  nxn  unit  matrix 


!  0 
v 


2; 


The  purpose  of  the  present  paper  is  to  give  sufficient  conditions  under  which 
(2.70)  has  a  solution  over  [0,  T], 
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2.3.2.  Derivation  of  the  Sufficiency  Conditions 
By  setting 


P(t)  =  P(T-t) 


(2.72) 


we  can  consider  equivalently  to  (2.70) 


P  =  F'P  +  P  f  +  Q  -  PS  P  -  PJ  S  PJ-JPSJ  P  +  J  PJ  SJ  PS 

o 

(2.73) 

P(0)  =  Kq,  t€[0,  T] 


whe  re 

X(t)  =  X(T-  t),  t€  [0, T] . 

If  P(t)  is  a  solution  of  (2.73)  on  [0,t')  where  t’  <  t,  then 

P(t)!|  <  0!(P(t)!!2  +  OflJP(t)||  +  q  (2.74) 

where  ![•'!  denotes  the  usual  sup  norm  of  a  square  matrix  calculated  for 
fixed  t,  and 

a  =max{2liA(t),l  ;  0  <  t  <  t] 

0=max{3lls(t)l!  +  !|So (t ) H ;  0  <  t  <  t]  (2.75) 

q  =max{  ]lQ(t)l|  ;  0  <  t  <  Tj 

The  a,  q  are  finite  due  to  the  piecewise  continuity  of  the  matrices  and  the 
finiteness  of  (0,  T]  .  Clearly  a,  0,  q  >  0.  We  assume  0/0,  since  if  0  =  0 
then  (2.70)  is  a  linear  differential  equation  and  the  solution  exists  for  T 
arbitrarily  large. 

Consider  the  scalar  differential  equation 
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y  =  +  ay  +  q  ,  y(0)  ~yQ>  t  >  0.  (2.76) 

4 

Using  Corollary  6.3  page  32  of  [16]  ,  we  obtain  that: 

if  y(0)  >  '|P(0)!|  and  y(t)  is  a  solution  of  (10)  on  [0,T]  then  the 
solution  of  (7)  exists  on  [ 0 , T]  and  |IP(t)ll  £  y(t),  t€  [0,T]. 


We  thus  conclude  that  a  sufficient  condition  for  the  existence  of  a  continuous 
solution  P(t)  of  (2.70)  over  [0,T],  is  that 

y(0)  >  I|kJ!  and  T  <  tf  (2.77) 

where  [0,  tf)  is  the  maximal  interval  of  existence  of  the  continuous  solution 
of  (2.76). 

A  straightforward  investigation  of  the  behavior  of  the  solution  of 
(2.76)  yields  the  conditions  under  which  (2.77)  is  satisfied.  We  state  the  results 
of  this  investigation  in  the  form  of  a  Theorem. 

Theorem  2.7:  Let  0  ^  0  and  set 


(i)  If  A  -  0,  and 


then  the  solution  of  (2.70)  exists  and 


(2.78.1) 


Although  Corollary  6.3,  page  32  of  [16],  is  stated  for  the  vector  case,  its 
extension  to  the  matrix  case  is  trivial. 
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1  2fi 

HP(t)H  S  Pl  +  c  -  3(T  -  C)  ’  c  "  Q-+  28||KoH  ’  [0’Tl' 


(ii)  If  A  >  0,  p2  <  s  0,  and 

!  11k  II  -  p. 


T  <  •—  in  | 


7a  '  fKo«-»l 


then  the  solution  of  (2.70)  exists  and 

8(Pl-p2)(T-t) 


i|P(t)ll 


Pl  -  P2Ce 


C  = 


1  -  Ce 

HKoll-pl 
llKoli  "  P2 


3(P1-p2)(T-t) 

t€  [0,T] . 


(iii)  If  A  <  0,  px  =  k  +  i\,  P2  =  k  -  iX,  k  =  ,  X  =  ,  i  =  >J~ 

i  -i  11K0I! " k 

T  <  ~T  [n  -  2  Tan  (—^t - )] 

"/W  x 

then  the  solution  of  (2.70)  exists  and 

l|P(t)l|  s  k  +  Uan  (X0(T-  t)+  C) 

,  l|K  ||  -k 

C  =  Tan  ( - - - ),  t€[0,T] 


where  Tan  *  is  the  inverse  tan  on  (-  ^  ,^). 


If  A  =  0,  Q.  =  0,  K.  =0  and  B.,  R. .  are  constant,  then  case  : 
i  i  i  ij  ’ 

and  T  can  be  taken  arbitrarily  large. 


*»* 


(2.78.2) 


(2.79.1) 


(2.79.2) 


1  and 


(2.80.1) 


(2.80.2) 


L  holds 


Consider  the  sets  U^,  and  defined  as  follows 


I  =  (t^tp  c  R,  fixed 


(2.81) 


E  =  [sjsCRnx  I,  S  open,  connected  and  projection  of  S  on  1=  1} 

in. 

U.  3  fu.lu.  :  S .  —  R  for  some  S .  €  Z,  u.  analytic  on  S.l,  i  =  1,2.  (2.82) 
i  l  i 1  i  i  i  i  iJ 

Uj  and  will  be  called  the  strategy  spaces.  Consider  also  the  fixed  time 

_  m.  m 

interval  [to,t_]  c  I  and  the  functions  f  :  Rn  X  R  XR  ^x  I  “  Rn,  :  Rn  -•  R, 
m.  m„ 

L.  ;  R  X  R  XRixI"*R,  i  =  1,2,  which  are  analytic  everywhere  in  all  their 
arguments. 

For  a  given  (u^  )  u^)  €  x  Uj  with  S.^0  S^E  we  consider  the 
dynamic  system 


x (t )  =  f(x(t),u1(x(t),t),u  (x(t),t),t) 

*<V  =  V  <Vto)€sins2>  rostostr 


(2.83) 


Definition  2.1:  A  pair  (u^ is  called  playable  at  (xo,tQ)  if 

S,DS-€E  and  the  solution  of  (2.83)  exists  over  [t  ,t,],  (It  will  neces- 
12  of 

sarily  hold:  (x(t) ,  t)€Sp>S2  te[tQ,tf].) 

For  a  related  definition  of  playability  see  [23].  If  (u^,up 
is  playable  at  (x  ,t  )  we  consider  the  functionals 

Cf 

j.(u.,u  )  =  g,(x(tf))  +  r  L  (x(t),u. (x(t),t),u  (x(t),t),t))dt 

1  Z  1  t  1  Z  (2.84) 

o 

i  =  1,2. 

•jjp 

Definition  2:2: A  pair  (u^Uj)?  U^x  is  said  to  be  a  Nash  equilibrium  pair 
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for  the  Nash  game  associated  with  (2.83)  and  (2.84)  on  a  set  S  =Sn(Rnx[t  ,tJ) 

o  or 

for  some  S€Z  if  and  only  if: 

(i)  it  is  playable  at  all  (x  ,t  )fS 


and 

(ii) 

~k  'k  */Jr 

J 1 C u  1 » u2  ^  *  J 1  ^ u  1 5  u2  ^  ’ 

V(u1,u2) 

playable  at 

(x  ,  t  ) 
v  0  0 

(2.85) 

it 

J2 Cui » ll2 )  s  J2^U1’U2^  ’ 

v(u1,u2)  eu^Uj 

playable  at 

(x  ,  t  ) 

0  o 

(2.86) 

for  all  (x  ,t  )  £  S  . 

o  o  o 

The  following  theorem  concerns  the  existence  and  uniqueness  of 
Nash  equilibria. 

Theorem  2.8:  Assume  that  there  exist  two  analytic'’  functions  u  and  u„, 

n  n  n  mi  2 

u^:R  XlXR  XR  -R  i  =  l,2,  such  that  u^  (y ,  s  ,q^ ,  q2)  and  u2  (y ,  s  ,q-^  ,q2  ) 

are  the  unique  global  solutions’’  of  the  minimization  problems 

minm  ^if  (y>ui’u2’s^ +Li^»ui»u2,s^  (2.87) 

uiPR  1 

where 

(y  i  s  ,  q1 ,  q2)  €  Rn  X  I  X  Rn  X  Rn  • 

Then  a  necessary  and  sufficient  condition  for  (u“,u^)  6 X  to  be  a  Nash 
equilibrium  pair  is  that 

3V.  (y ,  s)  3V  (y,s) 

u*(y,s)  =  Ui(y,s,  - ^ -  ,  - gp - )  i  =  l»2  (2'88) 


In  relation  to  this  see  for  example  [21],  p.  152. 
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where  V^(y,s),  V2(y,s)  are  the  unique  real  valued  analytic  solutions  of  the 
system  of  partial  differential  equations 


3V1  _ 


av1  av2 


!T+'s7  f(y’ui(y’s’  WW),u2(7,a’  w  a7)>s) 


av.  av  _  av,  av 

+  L.(y,u1(y,sJ  ~  ,  t~),u2  (y,s,  s)  =  0  i-1,2  (2-89) 


with  initial  values 


vi(y5tf)  =  gi(y)3  i-1,2,  V  y  €  Rn . 


(2.90) 


If  such  (Ul,u2)  exists,  then  it  is  unique  in  ^  x 

gropf :  Let  (u^upgl^x^  be  a  Nash  pair .  Then  the  functions 

Cf 

Vy’s)  =  gi(x(tf))+J  Li(x(t),u*(x(t),t),u?(x(t),t),t)dt  (2.91) 

s  z 

i-1,2 

where 

x(t)  =  f(x(t),u*(x(t),t),u*(x(t),t),t),x(s)  =  y,  t€[s,tf]  (2-92) 

are  analytic  in  y,s  (see  [20],  p.  44  and  [24],  p.  87,  Theorem  4.3)  and 
are  the  solutions  of  (2.89),  (2.90).  This  is  true  since  '2.89),  (2.90)  are 

just  the  Hamilton- Jacobi  partial  differential  equations  for  the  two  control 
problems  (2.85)  and  (2.86)  (see  [24],  p.  83,  Theorem  4.1  or  [22],  Theorem  1). 
sufficiency  part  follows  from  Theorem  4.4,  p.  87  of  [24].  The  uniqueness 
of  the  solution  of  (2.89)-(2.90)  within  the  analytic  class  is  an  immediate 
consequence  of  the  Cauchy-Kowalewsky  theorem  (see  [25],  p.  40).  C 


The 
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One  can  obtain  the  system  (2.89)-(2.90)  of  partial  differential 
equations,  under  assumptions  much  weaker  than  ours.  Nontheless,  the  results 
available  concerning  existence  and  uniqueness  of  solutions  of  general  systems 
of  partial  differential  equations  are  complicated.  Also,  they  usually  assume 
boundness  of  the  range  spaces  of  the  sought  solutions  and  of  the  domains  of 
the  independent  variables,  assumptions  which  we  did  not  make. 

2.4.2.  The  Linear  Quadratic  Case 

Next,  we  apply  Theorem  2.8  to  a  linear  quadratic  game.  Consider 

(see  [22]) 


x  -  Ax  +  BjUj,  +  B2u2  +  f(t),  x(tQ)  =Xq,  t  f  [tQ,tf] 


y,  ■  C.x,  y 


2  C2X 


Ji  =  2  (J  t(zi-yi)'Qi^i-yi)  +uiRiiui+ujRijujNt 

o 

+  X(tf)'  Kifx(tf)'}  ,  i  -  1,  2,  i  f  j. 


(2.93) 


where  A,  B^,  f,  C^,  Q^,  are  analytic  functions  of  t  over  all 

(t ',  tl)  and  Q.  =  Q!  >0,  R.  .  =  R.' .  >0,  R.  .  >0,  V  t  €[t  ,  t  J  . 

O  t  11  Ij  1J  —  11  o  f J 

=  >  o  are  constant  matrices.  All  the  matrices  are  assumed  to  be  of 

appropriate  dimensions. 


Corollary  2.1. Assume  that  K^t),  K2(t),  g^t),  g2(t),  §1(t),  $  (t)  are 
solutions  of  the  differential  equations 
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%  ■  Wl  +  Wl  +  K2S2K1 

-  K1S2K2  -  KiSUKJ  -  KjSljKi 

-  A'Kt  -  K.A  -  KlS.K.  -  C!  Q. 

k  ‘  KtSl8l  +  KlSl*t  +  K1S282  +  K2s28l 

-  KSlgi  +  KiSUgi  +  hSi}h>  -A'8i  'c;vi  +Ki£ 

<1  *  »{Vl  +  *2S28i  -  J  8iSi8i  -  SlVj  +1  *iVl  -  £  '8i 


(2.94) 


where 


W  =  Kif’  8i(tf)  =  °»  $i(tf)  =  °»  1  =  1’  2 


S.  =  B.rT^b:  i  -  1,  2 
i  i  ii  i 

S.  .  =  B.r'V  .R^B! ,  i  +  3,  i-»3  -  1,  2 

iJ  J  JJ  iJ  JJ  J 


S. .  =  0  i  =  j 

ij  J 


Then 


ui(x,t)  =  -R^BJ  (KjX  -  g.J,  i  =  1,  2 

constitute  the  unique  Nash  equilibrium  pair  in  U^xl^  for  the  Nash  game  as¬ 
sociated  with  (2.93),  for  any  (xo,tQ)S  Rn x  [t^t^]. 

Proof.  K^,  g^,  are  clearly  analytic  functions  of  t.  The  function 
V1(y,s)  “^-x'KjX  -  g|  x  +  $^  are  solutions  of  (2.89)  and  (2.90)  (in  the 
form  that  (2.89)  and  (2.90)  assume  for  the  problem  (2.93)).  Thus,  the 
previously  stated  Theorem  applies.  0 
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CHAPTER  3 

LEADER-FOLLOWER  STRATEGIES  AND  NONSTANDARD  CONTROL  PROBLEMS 

3.1.  Introduction 

In  the  present  Chapter  we  analyze  a  Leader-Follower  differential 
game  with  fixed  time  interval  and  initial  condition  xq  ,  where  the  leader 
has  current  state  information.  Thus,  the  strategy  space  of  the  leader  is  a 
space  of  functions  whose  values  at  time  t  depend  on  x(t),  x^  and  t,  i.e; 
u(t)  =  u(x(t),t;x  ).  Because  —-appears  in  the  follower's  necessary  conditions, 

O  d  X 

the  leader  is  faced  with  a  nonstandard  control  problem.  This  problem  is 
solved  here.  Actually  we  solve  a  more  general  class  of  such  nonstandard  con¬ 
trol  problems,  of  which  the  leader's  problem  is  a  special  case. 

The  structure  of  this  Chapter  is  the  following.  In  3.2  a  two-level 
L-F  differential  game  is  introduced.  This  game  leads  to  the  consideration  of 
a  nonstandard  control  problem  which  is  studied  in  3.3.  In  3.4  we  use  the  re¬ 
sults  of  3.3  to  study  further  the  game  of  3.2  and  in  particular  we  work  out  a 
linear  quadratic  l.-F  game.  In  3.5  the  relation  of  the  L-F  game  to  the  Principle 
of  Optimality  is  investigated. 

Two  Appendices  to  this  Chapter  are  given  at  the  end  of  the  Thesis. 

3.2.  A  Leader-Follower  Game 

In  this  section  we  introduce  a  two-level  L-F  game  and  show  how  it 
leads  us  to  the  consideration  of  a  nonstandard  control  problem.  This  non¬ 
standard  control  problem  falls  into  the  general  class  to  be  considered  in  3.3. 


Let 
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m  m 

U  =  [u|u:  Rn  x[tQ,tf]  "*  R  >  u(x,t)€R  for  x£  R  n  and  t€[tQ,tf], 
u^Cx.t)  exists  and  u(x,t),  ux(x,t)  are  continuous  in  x  and 
piecewise  continuous  in  t}  (3.1) 

m2 

V  =  {v|v:  [to>t^.]  -*  R  ,  v  is  piecewise  continuous  in  t}  .  (3.2) 

Consider  the  dynamic  system 


(3.3) 

(3.4) 

(3.5) 


where  u£  U ,  v€  V ,  x  is  the  state  of  the  system,  assumed  to  be  a  continuous 

function  of  t  and  piecewise  in  w.r.  to  t,  x  :  [tQ,t  ]  -Rn,  and  the  functions 
m.  m„  m.  m„ 

f  :  Rnx  R  iX  R  1  X  [  tQ,tf]  -Rn,  g,h  :  Rn  —  R,  L,M  :  R  X  R  X  R  L  X  [tQ,tf]  -R, 

are  in  w.r.  to  the  x,u,v  arguments  and  continuous  in  t.  The  u  and  v  are 
called  strategies  and  are  chosen  from  U  and  V  which  are  called  the  strategy 


spaces,  by  the  two  players,  the  leader  and  the  follower  respectively.  With 
the  given  definitions,  for  each  choice  of  u  and  v,  the  behavior  of  the  dynamic 


system  is  unambiguously  determined,  assuming  of  course,  that  for  the  selected 
pair  (u,v)  the  solution  of  the  differential  equation  (3.3)  exists  over  [tQ,t^]. 


Let  us  assume  that  a  LF  equilibrium  pair  (u*,v*)£lP<V 
exists.  For  fixed  uS  U,  Tu  is  determined  by  the  minimization  problem 
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minimize  J^CUjV) 

subject  to:  v£V  (3.6) 

x  =  f(x,  u(x,t),v,t),  x (tQ)  =  xqJ  t€  [tQ,tf] 

and  thus,  applying  the  Minimum  Principle  we  conclude  that  for  v €  V  to  be  in 
Tu,  there  must  exist  a  function  p  :  [tQ,t  ]  -*  Rn  such  that 

x  =  f(x,u,v,t)  (3.7.1) 


(3.8) 
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For  (A)  to  hold  it  suffices  for  example:  T-T1  on  We  conclude 

that  if  (A)  holds,  then  u*  is  a  local  minimum  of  the  problem 


minimize  J^Cujv) 

subject  to:  u£U,  v€T'u 


or  equivalently 


minimize  J^(u,v) 
subject  to:  u£U,  vf  V 


x  =  f  (x,u,v,t) 


(3.9) 

(3.10,1) 


-p-M  +uM  +  (f  +  u  f  )p 

X  X  U  XX  u 


M  +  f  p  =  0 

V 


x(to)  -  xo,  P(tf)  = 


d  h(x(tf ) 

ax 


(3.10.2) 

(3.10.3) 

(3.10.4) 


The  problem  (3.9)  is  a  nonstandard  control  problem  of  the  type  to  be  considered 
in  the  next  section,  since  the  partial  derivative  of  the  control  u  w.r.  to  x 
appears  in  the  constraints  (3.10)  which  play  the  role  of  the  svstem  differen¬ 
tial  eauations  and  state  control  constraints,  with  new  state  (x'jp1). 

Notice  that  the  leader  uses  only  x(t)  and  t  in  evaluating  u(x(t),t)  and  not 
the  whole  state  (x',p')';  i.e.  ,  the  value  of  u  at  time  t  is  composed  in  a 
partial  feedback  form  with  respect  to  the  state  (x1 ,p’)';  (recall  the  output 
feedback  in  contrast  to  the  state  feedback  control  laws).  If  one  were 
concerned  with  a  LF  game  composed  of  N  (>  2)  hierarchical  decision  levels 
([32]  ,  [33]),  then  the  leader  would  face  a  nonstandard  control  problem  where 


1 


See  Appendix  E. 
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the  N-th  partial  of  u  with  respect  to  x  would  appear. 

We  will  assume  that  the  state-control  constraint  (3.10.3)  can  be 
solved  for  v  over  the  whole  domain  of  interest  to  give 

v  =  S(x,p,u,t)  (3.11) 

where  S  is  continuous  and  in  w.r.  to  x  and  p.  This  assumption  holds  in  many 
cases,  as  for  example  in  the  linear  quadratic  case  to  be  considered  in  section 
3.4.  In  any  case,  direct  handling  of  the  constraint  (3.10.3)  by  appending  it, 
or  assumption  of  its  solvability  in  v,  does  not  seem  to  be  the  core  of  the 
matter  from  a  game  point  of  view.  However  the  following  remark  is  pertinent 
here.  Assume  that  we  allow  v^V,  where 

— 

V  =  [v|v  :  R  x  [tQ,t^]  -»  R  ,  v (x, t )  piecewise  continuous 

(3.12) 

in  t  and  Lipschitzian  in  x  where  x£Rn  and  t€  [to,tf]} 
instead  of  v£V.  The  assumption  of  solvability  of  (3.10.3)  will  again  give 

v(x,t)  =  S(x,p,u,t).  (3.13) 

Since  v(x,t)  will  be  substituted  in  the  rest  of  (3.10)  with  S(x,p,u,t)  from 
(3.13),  the  leader  will  be  faced  with  exactly  the  same  problem  as  after 
substituting  v(t)  with  S  from  (3.11).  Therefore,  no  additional  difficulty 
arises  if  one  allows  V  instead  of  V  and  assumes  solvability  of  (3.10.3) 

Substituting  v  from  (3.11)  to  (3.10)  we  obtain 

Cf_ 

minimize.  J(u)  =  g  (x(t^))  +  J  L(x,p,u,t)dt 

u€  U  to 


subject  to: 
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X 

Fx  (x,p,u, t) 

II 

_  p  _ 

F21(x,p,u,t)  +  uxF22(x,p,u,t) 

3h(x(t  ) ) 

x(tQ)  =  xo,  p(tf)  =  - — - 

where  L,  F^,  F^,  F22  stan<*  f°r  resulting  composite  functions, 
problem  (3.14)  is  a  nonstandard  control  problem  like  the  one  treated  in  Sec¬ 
tion  3.3  where  (x',p’)'  is  the  state  of  the  system. 

Besides  the  procedure  described  above  which  leads  to  the  considera¬ 
tion  of  the  problem  (3.14),  there  are  other  cases  in  which  such  problems  arise. 
For  example,  in  a  control  problem  where  the  state  x  is  available,  stochastic 

disturbances  are  present,  and  the  time  interval  [t  ,tj  is  very  large, 

o  r 

synthesis  of  the  control  law  as  a  function  of  x  and  t  is  preferable  over  a 
synthesis  not  using  x  (open  loop).  In  addition,  might  be  penalized  in 
the  cost  function  or  be  subjected  to  bounds  of  the  form  lux(x(t),t)j  <  K, 
t  €  [ tQ , t^] ,  where  K  >  0  is  a  constant . 

3.3.  A  Nonstandard  Control  Problem 

Consider  the  dynamic  system  described  by 

x(t)  =  f(x(t),u1(h1(x(t),t),t),u2(h2  (x(t),t),t),. . .  ,um(hm(x(t),t),t), 
u^(h1(x(t),t),t), . . . ,u”(hm(x(t),t) ,t) ,t) 
x(t0)  =  x0>  t €  [to,tf! 


(3.15) 
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and  the  functional 


J(u)  =  g(x(tf))+j  L(x(t)Ju1(h1(x(t),t))t),  .  .  .  ,um(hm(x(t),t),t), 
to 

u^(hi(x(t),t),t),...,u™(hm(x(t),t),t),t)dt 

.  .  ,  •  ,  n+m+mn+1  n  ,  n+m+mn+1  n  .  i  n+1  _^i 

where  the  functions  f:R  R  ,  L  :  R  -*R,h  R  “*  R 


(3.16) 


i  =  l,...,m,  g  :  Rn  -  R  are  continuous  in  all  arguments  and  in  with  respect  to 

the  x,  u,  u1.  The  functions  hi;Rn+'*'-R  ^ ,  are  continuous,  and  in  w.r.  to  x. 
x 

The  solution  x  of  (3.15)  is  assumed  to  be  continuous  and  piecewise  in 

C1  w.r.  to  t.  The  time  interval  [t  ,  tf]  is  considered  fixed  w.l.o.g.  (See  [35], 
page  27).  We  want  to  find  a  function  u  where 


u 


uL;RLx["t  ,  t  .1  —  R,  i  =  l,...,m 

or- 

u^(hL(x,t) ,t)  exists  and  u1 (hL (x, t) , t) ,  u^ (hL (x , t ) , t)  are  continuous  in  x 
and  piecewise  continuous  in  t,  for  x^R0,  tSCt^t^],  i  =  l,...,m  so  as  to 
minimize  J(u).  We  denote  by  U  the  set  of  all  such  u's.  Therefore  the 
problem  under  investigation  is 


minimize  J(u) 

subject  to  u € U  and  (3.15) 


(3.17) 


l  -  2 

The  restriction  h  c  C  w.r.  to  xis  somewhat  strong.  For  example,  the  case 
h(x,t)  =  x  if  to  s  t  s  t^,  h  (x,t)  =  0  if  t  <  t  s  t . ,  i.e.  the  state  is 
available  only  during  a  part  of  the  [to,  t^J  iS2not  included.  Nonetheless, 
it  can  be  approximated  arbitrarily  close  by  a  C  function,  like  any  function 
whicl^is  only  piecewise  C  .  Consequently,  from  an  engineering  point  of  view, 
h  ^  C  w.r.  to  x  is  not  a  serious  restriction. 


This  problem  is  posed  for  a  fixed  time  interval  [t  ,  t_]  and  a  fixed 

o  t 

•ff 

initial  condition  x  (t  )  *  x  .  Therefore  the  solution  u  ,  if  it  exists, 

oo 

will  in  general  be  a  function  of  tQ,  t^,  xq,  in  addition  to  being  a  function  of 
h(x,t),  t,  but  we  do  not  show  this  dependence  on  tQ,  t^,  xq  explicitly. 

We  use  the  notation 


df 

dL 

du1 

du1 

, 

,  mxn  matrix,  L  = 

• 

' 

u 

• 

df 

dL 

^  m 
ou 

_dum_ 

mxl  vector 


-  _  df 

fi - *  nxn  matrix,  i=l,...,m 

dCu1) 

x 


t  - 

- 7~  >  nxn  vector,  i  =  l,...,m 

S(ux) 


i  ,  i 


yj  1 

h  ~  (h^  ,  . . .  ,hj  , . . .  ,h_  )  ,  i  =  l,...,m,  j=l,...,q. 

4i  L 

u  i  =  (u1>*--,u^  ),  u1.  i=qi  x  qi  Hessii 


sian 


(3.18) 


y  y 


u1  -  £-h- .  (*,t)  .  du  (y  ,t) 

x  dx  i 

dy 

u  =  [u1  .'  ...  ;  •  m-> 


i  i  ,  nx 1  vector  i = 1 , 

y  =h  (x,t) 


>m 


m-, 
u  1 
x- 


nXm  matrix. 
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It  should  be  pointed  out  that  the  arguments  used  in  Classical 

Control  Theory  for  showing  that  for  the  fixed  initial  point  case,  it  is 

irrelevant  for  the  optimal  trajectory  and  cost  whether  the  control  value  at 

3 

time  t  is  composed  by  using  x  (t)  and  t  or  only  t,  do  not  apply  here  in 
general.  If  ul  *  u  (t),  t  6  [tQ.  t^]  ,  then  u^  =  0  and  this  changes  the 
structure  of  problem  (8).  Consideration  of  variations  of  u  is  also  needed 
and  this  was  where  the  previous  researchers  stopped,  see  [29].  This  problem 
is  successfully  treated  here.  We  provide  two  different  ways  of  doing  that, 
the  first  of  which  is  based  on  an  extension  (Lemma  3.1)  of  the  so-called 
"fundamental  lemma"  in  the  Calculus  of  Variations  (see  [36]). 

The  following  theorem  provides  necessary  conditions  for  a  function 
u  6  U  to  be  a  solution  to  the  problem  (3.17)  in  a  local  sense:  (we  assume  that 
U  is  properly  topologized) .  It  is  assumed  in  this  theorem  that  the  optimum 
u  has  strong  differentiability  properties,  an  assumption  which  will  be  relaxed 
later,  in  Theorem  3.2.  The  proof  of  this  theorem  is  based  on  the  following  Lemma. 


Lemma  3.1:  Let  M  :  [  tQ ,  t  ]  -  Rm,  IL  :  [  tQ ,  t  fJ  -  Rn,  i  =  1 ,  .  .  .  ,m,  y  :  [  tQ ,  t  fJ  -  Rn, 

be  continuous  functions,  such  that 

t  m  £  ■ 

j  M'<t)q>(y(t),t)dt+  Z  j  N’(t)c?  (y(t),t)dt  =  0 

t  1-1  t  y 

o  o 

for  every  continuous  function  cp  :  Rn  X  [  t  ,  t  ^  -  Rn,  where  c=  (cp^  ,  .  .  .  ,"m)  ,  and 

cp  is  in  C^  w.r.  to  y.  Then  M,  N,,...,N  are  identically  zero  on  t  ,tc~. 

I  m  "  o  r  - 

3 

This  holds  if  i)  the  set  of  the  admissible  closed  loop  control  laws  contain 
the  set  of  the  admissible  open-loop  control  laws  and  ii)  if  u*  is  an  optimal 
closed^loop  control  law  generating  an  optimal  trajectory  x*(t),  then 
v*(t)  ■  u*(x*(t),t)  is  an  admissible  open  loop  control  law. 
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Proof-,  The  choice*.  =  (0, . . .  .O,*1^, . . .  ,0)  \  *x:  [t  ,r  HR,*1 

±  or 

continuous  in  t,  i  =  l,...,m,  yields  Ms0  on  [t  ,t^].  Since  M  =  0,  the 

choice  cp.  =  (0, . . .  ,y  ,0, . . .  ,0)  ' ,  cp1  =  y  ,  where  ^  =  (t ,  >  •  •  .  )', 

1  tf  1  n 

t  :  [t0»t^] -Rn,  %  continuous  in  t,  results  in  J  N/ (t)f  (t)dt  =  0,  for  every 

fco 

such  f  ,  and  thus  IL  =  0  on  [t^t^]  is  proven  in  the  same  way  as  M  =  0  was 
proven.  o 


The  conclusion  of  the  above  lemma  holds  even  if  the  restriction 

.  k,  .  k  .  X. 

l  ii  m  i  . 

cp  (x,t)  -  •  • .  yn  *t  is  imposed,  where  k^ , . . .  ,k are  nonnegative 

integers,  since  the  polynomials  are  dense  in  the  space  of  measurable  functions 
on  ("t  ,t  ,1 . 

Theorem  3.1  :  Let  u*  €  U  be  a  solution  of  (3.17)  which  gives  rise  to  a  trajectory 

ri  -  {  (x*(t) ,  t)  I,  t  €  [to,tfl] ,  such  that  u1^  are  in  C1  w.r.to  x  in  a  n.b.d,  of 

{  (h  (x*(t),t),t),t  € [t  ,t_]}.  Then  there  exists  a  function  p  :  [t  ,tr] -Rn  such 

or  o  t 


•  m  ^i  i  i 

-p(t)  =  L  +f  p+  E  £  u.V  h.(L.+f.p) 
x  xr  i=l  j=l  J  xx  j  i  V' 

L  +f  p  =  0 
u  u 


(3.19) 

(3.20) 


V^h  (Li+fip)=0,  i  =  1, . . .  ,m 


(3.21) 


P(tf)  - 


Sg(x(tf)) 


(3.22) 


hold  for  t€[tQ,t^],  where  all  the  partial  derivatives  are  evaluated  at 
x*(t),  u1*(hL(x*(t),t),t),  u^*(h1(x*(t),t),t),t. 


68 


The  proof  of  this  Theorem  by  using  variational  techniques  and 
Lemma  3.1  is  simple  but  lengthy.  For  the  sake  of  completeness,  we  give 
it  in  Appendix  F. 

We  give  now  a  different  derivation  of  the  results  of  Theorem  3.1  , 
under  weaker  assumptions,  which  provides  an  interpretation  for  them  and  at 
the  same  time  an  extension  of  the  region  of  their  validity.  Let 

=  lu  I  u  :  [tQ,  t^]  —  R  ,  u  piecewise  continuous]  (3.23) 
Consider  the  problem 


minimize  JOu,^,  .  .  .  , um)  =  g(x(tf))  +/  Lfr.u.V^h  (x.t)^, .  .  .  ,'7xhra(x,t)um>t)dt 

subject  to  x  *  f  (x,u,V  h1(x,t)u1 , . .  .  ,7  hm(x,t)u  ,t),  x(t  )=x  ,  t€[t  ,t,] 

a  J-X  m  oo  o  z 

u€U  ,  u.  €  U  ,  i  =  1 ,  .  .  .  ,m.  (3.24) 

m  l 

Clearly,  if  J^,  are  the  infima  of  (3.17)  and  (3.24)  respectively,  it  will  be 
^1  —  ^2  '  Also ,  if  u  =  (u1 , . .  .  ,u  )  1 , u^ , . . .  ,um  solve  (3.24)  and  give  rise  to  3c ( t ) , 
then  a  u  =  (u  , . . . ,um) '  £  U  with 


u1(h1(x(t) ,t),t) 
um(hm(x(t),t),t) 

results  in  J2(u)  =  J(u,u15 . . . ,um)  and  gives  rise  to  the  same  x(t).  However, 
such  u  €  U  does  exist.  For  example  we  set 


=  u(t),  u*(hX  (x(t),t),t)=Vxh1(x(t),t)u.(t) 

i  = l,...,m  (3.25) 
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u1<h1(x,t))t)  =  a^(t)  hi(x,t)  +  bi(t)  (3.26) 

where 

ai(t)  =  “iCt)  (3.27) 

b^t)  -  GX(t)  -  a?  (t)hi (x(t) , t)  (3.28) 

i  -  1 , . . .  ,m 

This  u  satisfies  (3.25) .  Thus,  problems  (3.24)  and  (3.17)  are  actually  equivalent 
in  the  sense  that  for  each  given  (xo,tQ)  they  have  the  same  optimal 
trajectories  and  costs  and  their  optimal  controls  are  related  by  (3.25). 

The  conditions  of  Theorem  3.1  are  now  directly  verified  to  be  the 
necessary  conditions  for  problem  (3.24),  where  one  should  use  u  and  u^  in 
place  of  u  and  u\ respectively.  More  importantly,  the  conditions  of  Theorem 

y  * 

3.1  hold  if  one  considers  simply  u  Gu,  without  assuming  that  u  ^  is  in  C 

r  i  ,  y 

w.r.  toxinan.b.d.  of  i (h  (x* (t) ,t ) , t) , t € [ tQ ,t^] J .  This  weakens  the 

strong  differentiability  property  of  u*  assumed  in  Theorem  3.1  .  The 

relative  independence  of  u,  u1^,  was  exploited  in  proving  Theorem  3.1  , 

y 

when  the  special  form  of  the  perturbation  cp(y,t),  y'^(t)  (see  proof  of 
Lemma  3.1),  sufficed  to  conclude  (3.20)  and  (3.21).  This  independence  of  u  and 
u3’i  was  taken  a  priori  into  consideration,  when  problem  (3.24)  was  formulated. 

y 

Clearly,  even  if  higher  order  partial  derivatives  of  u  w.r.  to 


x  appear  in  f  and  L,  or  if  u,u  ^  are  restricted  to  take  values  within 

y 

certain  closed  sets,  the  equivalence  of  the  corresponding  problems  (3.17) 
and  (3.24)  holds  again  (with  appropriate  modifications  of  the  definitions 
of  U,  U^,  f  and  L) .  We  formalize  the  discussion  above  in  the  following 
theorem. 

Theorem  3.2:  Let  u  £u  be  a  solution  to  the  problem 


minimize  J(u)  =  g(x(t_))  +J  L(x,u,u1, . . .  ,um,t)dt 

L  ^  XX 

o 

subject  to:  x  =  f  (x,u,u^, . . .  ,um, t) ,  x(t  )  =  x  ,  t€[t  ,tJ 
'  X  X  o  o  o  f 


(3.29) 


u€v,  (u1(h1(x(t),t),t), .  .  .,um(hin(x(t),t),u11(h1(x(t),t),t) 

y 


umm(hm(x(t),t),t)')evo 


(3.30) 


where  V c  Rm  +  131,1  is  closed.  Then  there  exists 
o— 


p:  [t  ,tf]  “*  R  such  that 
°  i  _  q . 

m  j 


-p  =  L  +f  p  +  .E.  X  u^V  h^(L.+f.p) 
x  xr  i=l  j=l  j  xx  J  l  ir/ 


(3.31) 


L(x* (t) ,  u1*(h1(x*(t),t),t), . . . ,  um'f  (hm(x*(t),t),t),  u^*(h1(x*(t),t),t), 
um*(hm(x*(t),t),t),t)  + 

+  f’(x*(t),  u1(h1(x*(t),t),t), . . . ,  um*(hm(x*(t),t),t),  u1^  (h1  (x*  (t ) ,  t ) ,  t ) , 


,...,  um* (hm (x* (t ) , t ) , t ) :  p(t)  < 

<L(x*(t),  q*,...,q*,  Vxh1(x*(t),t)q1>...,Vxhm(x*(t),t)qm,t) 
+  £(x*(t),qj>...,q“,  Vxh1(x^(t),t)q1,...,Vxhm(xA-(t),t)qm,t) 
V(V....q>i,-,q;)eVo. 


(3.32) 
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P(tf)  = 


3g(x*(tf)) 

3x 


(3.33) 


for  t€[  t  ,  t ,]  .  c 

o  f 

It  is  remarkable  that  the  established  equivalence  of  the  problems 
(3.17)  and  (3.24)  refers  to  the  optimal  trajectories,  costs  and  control  values. 
It  does  not  refer  to  any  other  properties,  such  as  sensitivity,  for  example. 

It  is  thus  possible,  that  different  realizations  of  u1  (h1  (x,t) ,t)  other 
than  (3.26)  may  enjoy  sensitivity  or  other  advantages.  The  following  pro¬ 
position  provides  information  for  tackling  such  problems. 


Proposition  3.1. 


(i)  If  u  and  v  are  elements  of  U,  both  satisfying  (3.25),  so  does 


\u  +  (1-X  )u,  R  . 

(ii)  Let  m=l,  h1(x,t)  =  x^  and  x^,u,u^  be  scalarvalue  functions  of 
t,t€[tQ,t^] .  Then  the  function 

u(x,t)  *  e  u  (t)  +  [u1(t)-x1(t)u  (t)]  *  [x1-x1(t)] 


satisfies  u(x(t),t)  =  u(t),  ux(x(t),t)  =  TT^(t) 

(iii)  Let  x,  u,  u^  be  as  in  (ii).  Assume  that  the  scalar  valued 
functions  u(x,t),  v(x,t)  satisfy  u(x(t),t)  =  v(x(t),t)  = 

u(t)  and  ux(x(t),t)  =  vx(jT(t),t)  =u^(t).  Then  so  do  the 
_  ~2  2 

functions  */uv,y — — -  ,  assuming  that  u  and  v  are  properly 

behaved .  D 

The  proof  of  this  proposition  is  a  matter  of  straightforward  verification. 
The  assumption  in  parts  (ii)  and  (iii)  for  scalar  valued  quantities 
actually  induces  no  loss  of  conceptual  generality,  since  it  can  be  abandoned 
at  the  expense  of  increased  complexity  of  the  corresponding  expressions 
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of  course. 


The  nonuniqueness  of  the  solution  u  to  problem  (3.17)  is  obvious 


in  the  light  of  (3.25)  and  Proposition  3. 2. Nonetheless,  this  nonuniqueness 


is  a  nonuniqueness  in  the  representation  of  u1  as  a  function  of  h1  and  t. 


while  u|  ,  u1^!  are  the  same  for  all  these  representations.  The  non- 
7  I  if 

uniqueness  of  u|  ,u  J  ,  if  any,  can  be  characterized  in  terms  of  the 


possible  nonuniqueness  of  the  a^(t),  b^(t)  (see  (3.26)),  where  one,  w.l.o.g. 


restricts  u1  to  affine  in  h1  controls. 


One  very  basic  difference  between  problems  (3.17)  and  (3.24)  is  the  [1 


following.  It  is  clear  that  the  principle  of  optimality  holds  for  both  of 


these  problems,  in  the  sense  that  the  last  piece  of  each  optimal  trajectory 


is  optimal.  The  existence  of  a  closed  loop  control  law  (IT(x,t),u^(x,t), . . . , 
um(x,t)  which  results  in  an  optimal  solution  to  problem  (3.24)  for  every  initial 


point  (xo,tQ)  in  a  subset  of  R  is  guaranteed  under  certain  assumptions,  see 


[24].  A  corresponding  statement  does  not  hold  for  problem  (3.17),  i.e.  in  general 


there  do  not  exist  functions  u1  of  h1(x,t)  and  t  such  that  u=  (u1,...,um)'  i : 


an  optimal  solution  to  problem  (3.17)  for  every  initial  point  (x  ,t  )  in  a  subseti 

o  o  1 


of  R  .  This  can  be  easily  seen  to  hold  by  the  following  argument.  Let 


such  u  exist.  Then, 


(u1(h1(x,t),t),...,um(hm(x,t),t),u11(h1(x,t),t)',.. .,uram(hm(x,t),t)')' 

y  y 


is  a  closed  loop  control  law  for  problem  (3.24).  This  implies  that  there  must 


exist  a  solution  (u,u^, . . . ,um)  with  u—  (u^,...,u  )  of  the  partial  differential 


equation  of  Dynamic  Programming  associated  with  problem  (3.24)  which  satisfies 


u1  (x,t)  =  u1(hi(x,t),t)  and  —  =  ^h1  (x,t) -u.  (x,  t) ,  i  =  l,...,m,  which 
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is  not  in  general  true.  This  difference  between  problems  (3.17)  and  (3.24)  empha¬ 
sizes  the  fact  that  their  equivalence  holds  in  a  restricted  fashion,  i.e.  for  each 
initial  point  considered  independently  and  not  in  a  global  fashion,  like  a 
closed  loop  control  law  treats  the  initial  points. 

Two  final  remarks  before  entering  the  next  section  are 
pertinent  here.  First,  that  the  established  equivalence  of  the  problems 
(3.17)  and  (3.24)  reduces  all  questions  of  existence, uniqueness , controllability 
and  of  sufficiency  conditions  for  problem  (3.17) to  the  corresponding  ones 
for  (3.24).  Any  problem  of  the  form  (3.17)  where  terminal  constraints  and 
control  constraints  are  present  can  be  solved  and  necessary  and  sufficient 
conditions  can  be  written  down  in  as  much  as  this  can  be  done  for  the  problem 
(3.24)  with  the  corresponding  constraints  considered  in  addition.  Second, 


Theorem  3.2  still  holds  if  instead  of  the  initial  condition 

of  a  3  8  a '  3 ' 

x(tQ)  =  xq,  it  is  given:  x  (tQ)=xo  and  x  (t„)=xf,  where  x  =  (x  ,x  )'. 


In  this  case,  (3.33)  is  modified  to 


a  3g(x  (t  )) 

P  <tf)  =  - — §- 

d(x  ) 


S  Sh(x  <to>) 

and  p  (t  )  =  - 5 - 

°  3<x6) 


(3.34) 


where  the  more  general  cost  functional 


J  =  g(x  (tf))+  h(x (t0))+or  L(x,u,t)dt 

t 

o 


(3.35) 


is  considered  (see  [35]). 


Imposing  the  condition  —  =  V  h  .XI.,  where  XI  (x,t),u.  (x  t)  are  given  in 

3V(x  t)  °x  .  ?  1  1  1  1 

terms  of  — -  ,  where  V(x^t)  is  the  value  function  for  the  control  problem 

(3.24)  results  in  a  condition  that  V  must  satisfy  in  addition  to  being  a  solution 


of  the  dynamic  programming  partial  differential  equation. This  procedure  can 
be  used  to  single  out  a  class  of  control  problems  (3.17)  where  a  closed  loop 
control  law  exists. 
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3.4.  Solution  of  the  Leader  Follower  Game 

In  Chis  section  we  analyze  the  LF  Game  of  Section  3.2  by 
using  the  results  of  Section  3.3.  In  particular,  we  work  out  a 
Linear  Quadratic  LF  Game,  where  the  leader  is  paralized  for 

as  well. 

Let  us  consider  the  LF  Game  of  Section  3.2.  In  this  case, 
the  hNs  for  the  leader  (u) ,  are 

hi((x,p),t)  =  [I  0  J  [*]  =  x,  i=l,...,m 

nxn  nxir  p  i 

and  the  hl,s  for  the  follower  (v)  are  identically  zero.  Different  h  's  may 
be  used  to  model  different  information  structures  in  terms  of  x(t),  and  t 
available  to  the  leader  and  follower  at  time  t.  Thus  Theorem  3.2  is 
applicable  and  can  be  used  for  writing  down  the  leader's  necessary 
conditions.  From  the  results  of  the  previous  section,  we  conclude  that  the 
solution  for  the  leaders  u  -if  it  exists  -is  not  unique.  It  is  interesting 
to  notice  that  (3.26)  implies  that  the  leader  has  nothing  to  lose  if  he  commits 

himself  to  an  affine  in  x,  time  varying  strategy.  With  such  a  commitment, 
the  leader  does  not  deteriorate  his  cost,  does  not  alter  the  optimal 
trajectory,  and  also  the  follower's  optimal  cost  is  not  affected.  More 
noteworthy  is  that  the  affine  choice  for  the  leader  can  be  made  even  if 
f,  L,  M  are  nonlinear  and  u,  u^  are  constrained  to  take  values  in  given 
closed  sets.  In  addition,  v  may  be  constrained  to  take  values  in  a  given 
closed  set  in  which  case  (3.10.3)  should  be  substituted  bv  an  appropriate 
mequality.  In  accordance  with  the  discussion  in  the  previous  section,  we 
nave  that  in  general  there  does  not  exist  a  strategy  u  (x,t)  which  is  optimal 

'  ■verv  initial  point  (x  , t  1  in  a  subset  of  Rn  +  1 

o  o 
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Let  X  -  (X^X],)'  denote  the  adjoint  variable  for  problem  (3’.14)  with 
Xi»  X2  corresponding  to  x  and  p  respectively.  Then,  condition  (3.32)  results 
in 


[Mu(x,u,S(x,p,u,t) ,t) +  fu(x,u,S(x,p,u,t),t)p]X2  =  0 

Vt€[to,tf] 


(3.36) 


which  will  generally  make  the  leader's  problem  singular  [34].  This  is  to  be 
expected,  because  the  leader  exerts  his  influence  through  the  time  functions 
resulting  from  u  and  u  ,  which  are  actually  quite  independent,  and  u  is  not 

X  X 

penalized  or  subjected  to  any  constraint  in  the  initial  formulation  (2.3)— (3.5) . 
In  other  words  ,  the  leader  is  more  powerful  than  what  a  first  inspec¬ 
tion  of  the  original  problem  indicates.  One  way  to  restrict  the  leader's 
strength  or  to  avoid  the  singular  problem  could  be  the  inclusion  of  u^  in  L, 
i.e.,  L  =  L(x,u,u1, . . . ,um,t) ,  which  would  model  a  self  disciplined  leader, 

X  X 

or  to  impose  a  priori  bounds  on  ux»  for  example,  IJu^jl  s  K,  V  t£  [t^jt^] 
which  could  be  interpreted  as  a  constitutional  restriction  on  a  real  life 
leader. 


It  could  be  suggested  to  the  follower  to  penalize  u^  in  his 
criterion  while  u^  is  not  penalized  in  the  leader's  criterion.  This  would 
lead  to  the  appearance  of  u  in  (3.14)  (assuming  u*_  exists).  Thus,  in  ad- 


xx 


dition  to  (3.36)  a  similar  condition  due  to  u  appears  which  reinforces  the 

xx 

singular  character  of  the  problem.  If  the  leader  now  restricts  himself  to 


affine  strategies  in  x,  then  u  =  0  and  the  resulting  optimum  is  as  before. 


Actually,  the  leader  can  restrict  himself  to  a  quadratic  strategy  in 


(without  affecting  his  global  optimum  cost  and  trajectory)  having  thus  three 
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influences  on  Che  syscem,  namely  u,  u^,  u*  ,  from  which  only  u  is  penalized 
in  the  leader's  criterion-  -Therefore,  the  leader  will  do  better.  For  the 
follower  it  is  not  obvious  if  he  will  do  better  or  not. 

Let  us  work  out  a  Linear  Quadratic  LF  game.  The  leader 
is  penalized  for  u^  as  well,  by  including  it  in  L.  We  consider  the  dynamic 
system 


x  -  Ax  +  BjU  +  B2v,  x(tQ)  =  xqJ  t  6  [tQ,tf] 


(3.37) 


and  the  cost  functionals 


J1(u,v)  =  j[xjKlfxf  +  J  (x'Q1x+u,R11u  +  v’r12v+  £  u^  R^u^dt]  (3.38) 

t  i=l 


J2(u»v)  =  j[x^K2fxf  +  J  (x'Q2x+u'R21u  +  v'R22v)dt  ] 

fco 


(3.39) 


where  the  matrices  A,  B.,  Q_.  ,  R.  . ,  R.  are  continuous  functions  of  time  and 

i  T.  ij  x 

Q. ,  R. . ,  R.  >  0,  R, ,  >0  are  symmetric.  R„„  >  0  is  nonsingular  V  t(  [t  ,tf]  , 
i  ij  i  11  22  u  r 

which  guarantees  (3. 11). The  follower's  necessary  conditions  are  (recall  (3.10)) 
v  =  -R^2B2P  (3.40) 


x  *  Ax  +  B^u  -  ^2^22^  p 
P  =  -Q2x  -  UxR21u  -  A'P  -  UxB;P 

=  xo’  "  Sf^f* 


(3.41) 

(3.42) 

(3.43) 


Therefore,  the  leader's  problem  is  (recall  ('3'. 9)  ,  (3.14))' 


3We  assume  that  Assumption  (A)  holds.  See  also  Appendix  E. 
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l  f 

minimize  J(u)  =  — [x' K  x,  +  f  (x'Q ,x+u'r  u  + 

L  I  11  f  v  t  1  11 

o 

subject  to: 

•  1 
x  =  Ax  -  +  BjU 

P  -  "Q~x  -  A'p  -  u  B'p  -  u  R-.u 
2  r  x  lr  x  21 


(3.44) 


x(tQ)  =  xo,  p(tf)  =  K2fXf. 


(3.45) 


(3.46) 


(3.47) 


The  necessary  conditions  for  the  leader  in  accordance  with  Theorem  3.2  are 
(3.45),  (3.46),  (3.47)  and 


Ruu  +  b;x1  -  f2lu-x2  .  0 

tRiui  :•••:  +  >-2(r21»+b;p)’  -  o 

X1  ~  "^1X  ”  A'^l  +  Q^2 

X2  ~  ~B2R22R12R22B2P  +  ^^^l  +  +  BluxX2 

^l(tf)  -  Klfxf,  = 

For  simplification  we  assume  further  that 

Ri  =  YiIj  Yi  =  Y  >  0,  i  *  1,  ....  mx 


R11  ‘  ■’  R22  '  1 


and  (3.48),  (3.49)  are  easily  solved  for  u  and  u  to  yield 


(3.48) 

(3.49) 

(3.50) 

(3.51) 

(3.52) 


(3.53) 


1 
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u 


R2iR2iJ 


■t^l 


(3  54) 


u 

X 


1 

Y 


(3.55) 


which  can  be  substituted  into  (3.45),  (3.46),  (3.50),  (3.51)  to  yield  a  nonlinear 
system  of  differential  equations,  with  unknown  x,  p,  and  boundary  condi¬ 

tions  (3.47)  and  (3.52).  If  y  -+  +00,  then  (3.54)  and  (3.55)  yield  u  -»•  0  and 
u  -*  -B-J^  ,  and  thus  the  solution  tends  to  the  open  loop  solution,  i.e., 

u  -  u(t)  v  *  v(t),  as  the  resulting  form  of  (3.45),  (3.46),  (3.50),  (3.51) 
indicates  for  y  -*■  ([27], [28]). 


3.5.  Relation  to  the  Principle  of  Optimality 

It  has  been  shown  in  [29]  through  a  counterexample  that  the  Principle 
of  Optimality  does  not  hold  for  LF  games.  To  make  this  statement  more 

precise,  let  us  assume  that  the  LF  problem  of  Section  3.2  has  been  solved 

★  * 
in  [t  ,  t_]  and  x  is  the  optimal  trajectory.  While  the  process  is  at  (x  (T) ,f) , 
o  f 

where  t  ^  tf  ^  t,,  we  stop  and  solve  the  same  LF  game  on  [t,  t  ] 
of  t 

—  *  —  — * 

with  initial  condition  x  (t)  =  x  (t).  Let  x  be  the  optimal  trajectory  for 

— * 

the  second  problem.  Then  x  does  not  have  to  coincide  with  the  restriction  of 
*  — 

x  on  [t,  t^] .  The  explanation  is  the  following.  The  leader  is  faced  with  the 
control  problem  (3.14)  which  has  boundary  conditions  x(t  )  =  x  and 

*  (*  ctf»  ,°  *  0 

p  (t^)  a  -  ,  given  at  t  and  t^.  Let  (x  ,  p  )  be  the  optimal 

trajectory  of  this  problem.  If  the  leader  is  asked  to  solve  the  same  control 

*  — 


problem  on  [t,  t^]  with  boundry  conditions  x  (t)  =  x  (t)  and  p  (t^)  =  - g 

—  ★  _ „ 

there  is  no  necessity  for  p  (t)  ■  p  (t).' 


Sh  (x  (tf)) 
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Even  more,  if  X^,  are  the  adjoint  variables  of  the  leader's  control  problem 
on  [t  ,  t  ]  and  X  ,  X  are  the  adjoint  variables  of  the  leader's  control 

O  £  1  Z 

problem  on  [t,tf],  corresponding  to  x  and  p  respectively,  it  will  be 
3g  (x  ttf))  _  '  3g  (x  (t  ))  _  _ 

\  (tf}  =  ^ - ’  S  (to>  =  °’  \  ‘V  =  - - ’  X2  (C)  =  °’ 

If  the  P'inciple  of  Optimality  were  holding,  it  should  be  X (t)  =  X ^  (t)  =  0, 

which  is  not  true.  Actually,  X  (t)  =  0,  Vt£  [t  ,t,l  is  a  necessary 

i  of 

condition  for  the  Principle  of  Optimality  to  hold  The  condition 

X2  (t)  *  0,  [tQ,  t  ]  can  be  used  for  deriving  more  explicit  conditions 

in  terms  of  the  data  of  the  problem  for  the  Principle  of  Optimality  to  hold. 

Let  us  consider  the  linear  quadratic  game  of  Section  3.4.  As  it 

was  shown  in  the  previous  paragraph, X  (t)  =  0  Vt€  [t  ,  t  ] ,  is  a  necessary 

l  of 

condition  for  the  Principle  of  Optimality  to  hold.  With  X^  s  0,  (3.51)  yields 

-82^2Rl2S2B2>'  +  B2^2B?1  '  0 

from  which,  by  assuming  rank  B2  =  n^,  we  obtain  equivalently 

-Ki2i£b;p  +  B^1  -  0. 


Also,  (3.49)  yields 


u  =0,  i  =  1, . . , ,m, . 
x  1 


(3.56) 


We  conclude  that  under  the  assumption  rank  =  m  ,  (3.45)-(3.52)  simplify 


to  give 


x  =  Ax  +  B^u  +  B2v 


(3.57) 


xi =  ■  A'X1 


(3.58) 
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Ruu  +  B{^1  =  0,  R^v  +  B^l  =  0 
x(to)  =  xo>  \(tf)  = 
p  *  -Q2x  -  A'p 

v  *  -R^2B2p 

p(Cf)  =  K2fxf. 


(3.59) 

(3.60) 

(3.61) 

(3.62) 

(3.63) 


(3.57)-(3.60)  show  that  the  leader's  problem  can  be  considered  as  a  team  problem 
under  the  "constraint"  (3.56),  with  optimal  solution,  say  (u*,v*)  and  (3.61)  - 
(3.63)  show  that  the  same  v*  must  be  the  follower's  optimal  reaction  to 

■k 

the  leader's  choice  u  .  Actually  (3.56)  is  not  at  all  a  constraint,  since 
with  ^2  s  0,  (3.46) /where  appears)  is  not  really  considered  by  the  leader. 

So,  the  leader  operating  under  (3.45)  and  wanting  to  minimize  (3.38)  may  as  well 
choose  u^  =  0,  since  he  is  penalized  for  u^,  while  u^  does  not  appear  in  (3.45). 

The  same  analysis  and  conclusions  carry  over  to  the  more  general 
game  of  Section  3.2  (see  (3.1)— (3.5)  and  (3. 11) ), since  the  condition  =  0  on 
[t^,  t^]  comes  from  the  demand  that  the  transversality  conditions  hold 
Vt€  [t^,  t^]  and  is  not  affected  by  the  fact  that  in  (3.4)  u^  is  no-  penalized. 
Notice  that  if  the  leader's  cost  functional  (3.4)  is  substituted  by 


•^(UjV)  =  g(x(t  ))  +  j  [L(x,u,v, t)  +  I  u'R  ul1dt 


(3.64) 


then  (3.56)  holds  again. 
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The  idea  behind  the  condition  =  0  on  [tQ,t^]  is  that  the  leader 
is  not  really  constrained  by  the  follower's  adjoint  equation  and  therefore 
the  leader's  problem,  being  independent  of  the  follower's  problem,  becomes  a 
team  control  problem. 

In  conclusion,  a  necessary  condition  for  the  Principle  of  Optimality 
to  hold  for  the  LF  games  of  Sections  3.2  (and  3.4),  is  that  the  leader’s 
problem  is  actually  a  team  control  problem.  But  for  a  control  problem  with 

fixed  initial  conditions,  the  Principle  of  Optimality  does  hold.  We  thus 
have  the  "if  and  only  if"  statement:  The  Principle  of  Optimality  holds  for 
the  problems  of  Section  3.2  (and  3.4)  (see  (3.1)-(3.5),  (3.11)  and  (3. 37 ) — ( 3 . 39) 
respectively)  if  and  only  if  the  leader's  problem  is  a  team  control  problem 
for  both  the  leader  and  follower. 
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CHAPTER  4 

SUFFICIENT  CONDITIONS  FOR  LEADER-FOLLOWER  AND  NASH 
STRATEGIES  WITH  MEMORY 


4.1.  Introduction 

In  the  present  Chapter  we  consider  a  continuous  time  deterministic 
differential  game  with  a  linear  state  equation  and  quadratic  cost  functionals. 

We  consider  the  case  where  the  players  have  at  each  instant  of  time  recall  of 
previous  values  of  the  trajectory,  i.e.  they  have  memory.  Both  LF  and  Nash 
equilibrium  concepts  are  considered  and  sufficient  conditions  are  developed 
for  a  class  of  problems  to  have  solutions  affine  in  the  information.  Particular 
emphasis  is  placed  on  the  LF  case. 

The  structure  of  this  Chapter  is  the  following.  In  4.2  we  give  an 
example  of  an  LF  game  where  the  leader  by  using  previous  values  of  the  state 
forces  the  follower  to  such  a  reaction  that  the  leader's  final  cost  is  the  same 
as  it  would  have  been  if  both  leader  and  follower  were  striving  to  minimize 
the  leader's  cost.  The  main  steps  in  solving  this  example  serve  as  an  illustra¬ 
tion  of  how  a  more  general  case  should  be  analyzed.  In  4.3  we  derive  sufficiency 
conditions  for  optimality  for  a  control  problem  of  a  special  form  (of  interest 
on  it's  own)  which  are  used  in  the  next  sections.  In  4.4  we  apply  the  results 
of  4.3  to  a  LF  game  where  the  leader  has  recall  of  the  previous  trajectory  and 
the  game  is  such  that  the  solution  of  the  LF  game  (u*,v*)  minimizes  the  leader's 
cost  over  all  admissible  (u,v) ,  i.e.,  the  leader's  problem  is  actually  treated 
as  a  team  problem  of  both  the  leader  and  follower.  In  4.5  we  consider  certain 
special  cases  and  generalizations  of  the  LF  game  of  4.4.  In  4.6  we  apply  the 
results  of  4.3  to  a  Nash  game  where  the  two  players  have  perfect  recall  of  the 
whole  previous  trajectory.  One  Appendix  to  this  Chapter  is  provided  at  the  end 


of  the  Thesis. 
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4.2.  Introductory  Example 

In  this  section  we  provide  an  example  of  an  LF  differential 
game  where  the  leader  uses  the  previous  values  of  the  state  in  calculating 
his  control  values.  The  game  considered  is  such  that  the  leader,  by  using 
this  type  of  strategy  forces  the  follower  to  such  a  reaction  that  the 
leader's  optimal  cost  is  the  one  he  would  achieve  if  both  leader  and  follower 
had  as  their  common  objective  the  minimization  of  the  leader's  cost;  i.e., 
the  leader's  problem  to  minimize  is  actually  treated  as  a  team  problem 
where  the  team  is  composed  by  both  the  leader  and  the  follower.  A  similar  idea 
is  used  implicitely  in  [44].  The  strategies  found  provide  an  LF  equilibrium 
pair,  with  the  property  above,  for  any  xq  .  Also,  the  dependence  of  the 
leader's  control  values  on  previous  state  values  is  not  trivial  in  the  sense 
that  the  same  result  -  team  solution  of  the  leader's  problem  -  cannot  be 
achieved  by  strategies  depending  only  on  current  state  value's  information. 

We  develop  the  example  in  such  a  way  that  the  proof  of  the  optimality  of  the 
indicated  strategies  is  clear.  Actually  we  do  not  give  only  one  example  but 
provide  a  way  of  constructing  a  whole  class  of  Stackelberg  games  with  the 
above  properties. 


Consider  the  following  state  equation  and  cost  functionals 


x  =  2x+u+v  ,  x (0)  =  xq,  t  €[ 0, 1] 


Jj  =  4x(l)2  (6x2  +u2  +v2)dt 

=  2x(l)2  +  J  (q  x2  +rv2)dt 


(4.1) 

(4.2) 


4.  ...  — ■ 


(4.3) 
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where  x,  u,  v  are  scalar-valued.  The  solution  of  the  problem 


minimize  J, 


subject  to  (4.1) 


(4.4) 


u  =  -2k  x,  v  =  -2kx 


(4.5) 


where  k  solves  -k  =  3 +4k-4k  ,  k(l)=2,  and  is  given  explicitly  by 


k(t)  = 


15  +e8 

10-2e8<C-1> 


We  want  to  show  that  there  exist  q,  r,  4  so  that 


the  problem 


minimize  J, 


(4.6) 


subject  to  x  =  2x  +J&jX  +l^z  +  v  ,  x(0)=xo 

z  =  x  ,  z(0)  *  0 


(4.7) 


has  the  solution 


v*  =  HjX  +  p^z 


(4.8) 


and  that 


j4x(t)x*(t) +j22(t)z*(t)  =  -2k (t)x* (t)  ,  t€[0,l] 

v*|  =  p,1(t)x*(t)  +u2(t)z*(t)  =  -2k(t)x*(t) ,  t€[0,  l] 


i2(t)  a  0  t€[ 0, 1] 


(4.9) 


(4.10) 


(4.11) 


for  any  xq  where  x*  is  the  common  optimal  trajectory  of  the  problems  (4.7)  and 
and  (4.4)  since  (4.9)  and  (4.10)  will  hold. 
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It  is  clear  that  if  conditions  (4.9)  3nd  (4.10)  are  satisfied,  then 

ft 

the  pair  (u  =  (t)x (t)  + (t) j  x(T)dr,  v  = -2k (t)x (t) )  constitutes  an 
1  z  o 

LF  equilibrium  pair  for  the  LF  differential  game  associated  with 
(4.1)-  (4.3)  and  where  U  = [u| value  of  u  at  time  t  is  given  by  u(x  ,t),  where 
xt£C([0,t]  ,R) ,  xt(0)=x(9)  V9€[0,t]  ,  u(xt>t)  is  Frechet  differentiable  in  x 
and  piecewise  continuous  in  t£[0,l]},  V={v|v  is  a  function  of  x(t)  and  t, 
at  time  t,  v(x,t)  is  continuous  in  x€R  and  piecewise  continuous  in  t€[0,l]l. 

We  set 

Qf(t)  =  2  -  4k (t)  (4.12) 


and  thus  the  optimal  trajectory  for  the  problem  (4-4)  is 

t 

J  a  (t )dT 

x*(t)  =  e°  *  Xq 

t  j"W)da 

z*(t)  =  £  [e°  ^dT'x0 

The  solution  v*  of  (4.7)  is 

v*  =  •  T"  (P!x+P2z) 

where 


-F1!  =  2(2+^!>P!  +  2p2  +  q  - 

1  2 

T  pl 

»  Px (1)  =  2 

-P2  =  i2Pl  +  +,Z1^P2  +  P3  ~ 

~T  P1P2 

>  P2d)=0 

*^3  ~  ^^2P2  ~  p2 

»  P3(1)=0 

Substituting  (4.13)  and  (4.14)  in  (4.9),  (4.10)  we  obtain 

?!  =  2r  k  -  p2cp 
■&!  =  -2k  - 


(4.13) 

(4.13) 


(4.14) 


(4.15) 


(4.16) 
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where  .  r  a(c)dc  T  a(T)d'r 

cp(t)  =  (f  e°  dT)(e°  ).  (4.17) 

0 

(Since  9  =  z/x,  it  is  easy  to  see  that  cp  =  2- (2-4k)cp. )  Substituting  from 

(4.16)  into  (4.15)  we  obtain  further 


2£^rk  -  (p^  i-ccp^  ~  q  +  4rk2  +  6r  -  2rk  =  0 

-P2  =  X2(2rk-p2cp)  -  (2-2k-22cp)p2  +  P3  *  ^  (2rk-p99)P2 
12 

-P3  =  2i2P2  --  p2 
2r  (l)k(l)  -  p2  (l)cp(l)  =  2 
P2  (1)  -  0 
P3(1)  -  0. 

From  (4. 19)and(4. 20), setting  w  =  p2+9p3  we  obtain 


(4.18) 

(4.19) 

(4.20) 

(4.21) 

(4.22) 

(4.23) 


w  =  -2rkX2  -  (2-4k)w,  w(l)  =  0.  ‘  (4.24) 

Solving  (4.18) for  P2+CCP3  and  substituting  in  (4.24)  we  obtain  finally  the 
following  system  equivalent  to  (4 . 18) - (4 . 23) 


l2cprk  +  X2[rk+2(l-2k)9rk+^  (c?rk)]  + 

+  [■—  (2rk2  +3r-fk)  +2  (l-2k)  (2rk2  +  6r-2rk)J  +[  (l-2k)q  -Jq]  =  C 

(4.25) 


-P3  =  2i2(w-9p3)  --  (w-<PP3)2  (4.26) 

2 

w  =  2j&29rk  -  q  +  4rk  +  6r  -  2rk  (4.27) 

P2"w-q,p3  (4.28) 

Jt2(l)  =  %(-ll  +4f(l)  +q(l))o(l)_1  (4.29) 

P3(l)  =  0  (4.30) 

r(D  =  %• 


(4.31) 


We  can  choose  now  r,  q,  J A  ^  as  to  satisfy  (4 . 25)- (4 . 31)  and  (4.16).  We  choose 
r(t)  to  be  a  twice  differentiable  function  of  tG [0,1]  with  r(t)>0,  r(l)=y  and 
q(t)  to  be  a  differentiable  function  of  t.  Obviously  q  and  r  can  be  chosen 
so  that  the  linear  differential  equation  for  A  (4.25)  with  initial  condition 
(4.29)  has  the  solution  &  (t)  *  0.  For  example  let  r  =  -j,q  =  constant  ^  11. 
Notice  that  the  differential  equation  (4.25)  for  can  be  solved  explicitly 
for  JL  as  soon  as  r  and  q  are  specified  since  cp  and  k  are  known.  Nonetheless 
since  cp(0)=0,  the  point  t=0  is  a  singular  point  of  this  differential 
equation.  The  singularity  was  sort  of  expected  to  appear  since  as  it  has  been 
shown  in  Chapter  3,  the  leader's  problem  is  singular  with  respect  to  the 
partial  derivative  of  his  control  and  arguments  similar  to 

those  in  Chapter  3  can  be  used  to  show  that  this  holds  even  for  the  case  where 
u  is  allowed  to  be  of  the  more  general  form  u(xt>t).  Notice  also  that  the 
only  essential  restriction  on  the  follower's  cost,  in  order  for  the  leader 
to  achieve  his  team  solution  (allowing  even  A^=0)  is  that  r(l)=%. 

If  the  leader  were  allowed  to  use  a  strategy  u(x,t)  which  is 
perhaps  nonlinear  in  the  current  state  x(t),  but  he  were  not  permitted  to 
use  previous  values  of  the  state,  then  it  should  again  be  true  that 
u  (x*  (t) , t)  = -2k  (t)x*  (t)  for  every  xq,  i.e. 

rta(T)dT  ^’ta(T)dr 

u(e°  x  ,t)  =  -2k(t)e  x  Vx  €  R 

o  '  o  o 

from  which  we  obtain  that  u  is  linear  in  x.  Therefore,  we  conclude  that  for 
the  given  example,  if  the  leader  wishes  to  achieve  his  team  solution  (for 
any  x  )  when  he  applies  his  strategy  and  cannot  do  that  with  a 
linear  strategy  in  x(t),  he  cannot  do  it  with  a  nonlinear  in  x(t)  strategy 
either.  Therefore,  use  of  memory  is  his  only  way  to  achieve  his  team  solution. 
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In  Che  example  presented  here  the  two  crucial  steps  were  the 

identifications  (4.9),  (4.10)  and  the  use  of  the  fact  that  the  conditions  (4.14), 

(4.15)  are  sufficient  to  characterize  completely  the  optimal  reaction  of  the 

t 

follower  to  the  leader's  strategy  u  =  JL  (t)x  (t)  +l~  (t) J  x (T)dT .  Therefore, 

L  L  6 

in  order  to  generalize  the  procedure  presented  to  cases  where  more  general 
types  of  strategies  are  used  by  the  leader,  one  should  provide  sufficient 
conditions  for  the  problem  faced  by  the  follower,  in  addition  to  imposing 
identifications  similar  to  (4.9)  and  (4.10).  In  the  next  section  we  prove 
sufficiency  conditions  for  a  special  type  of  control  problem,  which  we  will 
use  later  in  guaranteeing  the  optimality  of  the  follower's  reaction,  when 
the  leader  uses  strategies  represented  as  continuous  linear  functionals 
over  the  whole  previous  trajectory. 


4.3.  A  Control  Problem  With  State-Control  Constraints. 


Consider  the  problem  (P) : 


minimize  J  =  \  [x?(tf )Fx(tf)  +  J  (x'  (t)Q(t)x(t)  +  u' (t)R(t)u(t) )dt] 

(4.32) 


subject  to:  x(t)  -  A(t)x(t)  +  B(t)u(t),  x(t^)  Xq 

J  [d  T)(t,s)]x(s)  +  J*  [dgTl1(t,s)]u(s)=  q(t) 


(4.33) 

(4.34) 


u€Ln 


where  the  matrices  A,  B,  Q  =  Q'  2  0,  R  =  R1  2  0,  are  piecewise  continuous 
functions  of  time,  x  (t)  gRn,  u(t)  €Rra,  and  the  interval  [tQ,tf]  the  matrix 
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p  =  y'  >  o  rand  q  6  l.  ,  are  fixed.  The  solution  x(t)  of  (4.33)  is  assumed 

1 ,  tc 

to  be  absolutely  continuous,  so  that  (4.33)  holds  almost  everywhere  with  respect 
to  the  Lebesgue  measure  in  [tg,t^].  The  integrals  in  (4.34)  should  be 
interpreted  as  Lebesgue -Stieltjes  integrals.  The  matrix  valued  function 
T](t, 9) ,  T\ ;  [t  ,t^l  X  R  "  is  measurable  in  (t,9),  normalized  so  that 

f 0  ,  for  9  a  t 

^(t,9)  =  m  (4.35) 

(jKt.tg),  for  9  £  tQ  . 


TJ( t ,  ® )  is  continuous  from  the  left  in  9  on  (t^,t^), 
variation  in  9  on  [tg,t^]  for  each  t,  and  there  is  a 


||  J  [dsTKt,s)]9(s)||L  *  c (t)  1|  9  ||c 
C0  1 


Tl(t,9)  has  bounded 
c  6  L  such  that 

1  9  1 

(4.36) 


for  all  t  €  [tg,tf]  and  for  all  9  €  C^.  Exactly  the  same  assumptions  hold 
for  :  [tg.tjJ  X  r  -  r^3™1  with  c^  replacing  c  in(4.36).T|  and  are  given 
for  the  problem  (P).  The  dimension  k  is  arbitrary  but  fixed. 

Problem  (P)  is  of  interest  to  us  since  we  will  use  the  results 
of  this  section  in  the  next  ones  where  we  will  consider  games  with  delayed 
information  structure.  Nonetheless  it  is  of  interest  on  it's  own.  It  is 
worthy  to  point  out  that  (P)  is  of  a  quite  general  form,  since  for  example, 
the  problem  (P') 

t£ 

minimize  j  [x ' (tf )Fx(tf )+f  (y' (t)Qy(t)+u^(t)R(t)u1(t))dt] 

^ o 
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y  (t)  =  J*  [d  Tl3(t,s)]x(s) 

t 

o 


^l(t)  =  f  [dgTl4(t)s)]u(s) 


X  (t  )  =  X 

o  o 


can  be  brought  to  the  form  (P)  by  introducing 


r  . 

u,(t)  =  J  [d  11  (t  ,  s )  ]  x  (s  ) 


2 '  '  -i  -  s 
o 


u-j(c)  *  J  [d  Tl2(tJs)]x(s) 

J  t  8 

o 


u4(t)  -  y (t) . 


C4.37) 


(4.38) 


Using  (4.37),  (4.38),  (P')  can  be  written  equivalently 
1  Cf 

minimize  j  U'(tf)  Fx(tf)  +J  (u^Qu4  4-u^Ru^dt] 

*"0 

(4.39) 

subject  to  x(t)  *  u2(t)+u3(t) 

Cf  ^  tf  Cf 

ux  -  J*  [dll'  1»,  u2  =  J  [dll^x,  u3  -  J  ( dTj2]  x ,  u4  -  J  [dTl3]  x  (4.40) 

C°  Co  Co  to 

where  the  role  of  x  and  u  in (4.33),  (4.34)  is  played  now  by  x  and 

(u> >u2 »u3 >u4)  respectively.  Clearly  (4.40)  is  of  the  form  (4.34). 

In  the  following  theorem  we  give  sufficiency  conditions  for  optimality 

for  the  problem  (P).  The  proof  is  carried  out  by  reformulating  (P)  as  a 

constrained  optimization  problem  in  a  Banach  space  and  is  given  in 


Appendix  G. 
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Theorem  4.1:Consider  the  problem  (P)  and  assume  that  there  exist  functions 

-*Rn,  \  x*:  [t0,tf]-Rn,  u*€Loo  m>  where  t*  is  of  bounded 

variation  on  [tg,t^]  and  continuous  from  the  right  on  (tg,tj),  and  x*  is 
absolutely  continuous,  which  satisfy  (4.33),  (4.34)  and 


f  f 

-  J  (R(  t)u*(t)  +  B'(-r)u(T))dT  +  f  i;(r,t)\(T)dT  =  0 

A.  i.  * 


(4.41) 


'0 
t , 


- f  ~f 
M'(t)  -  J  (Q(t)x*(t)  +  A,(T)n(T))d-r  +  j  Tl' (T,t)\(T)dT  =  Fx(t  )  (4.42) 

'o 

Then  u*,  x*  solve  (P)  . 

It  is  easy  to  see  that  in  case  T|,  ^=0,  then  (4.41)  and  (4.42) 

reduce  to 


R(t)u*(t)  +  B'(t)Kt)-0 


-KO-  Q(t)x^t)+A’(t)Kt),  Ktf)=?x(tf) 


as  it  should  be  expected. 

Theorem  4.1  can  be  easily  extended  to  the  case  where  cross  terms  u'Lx 
exist  in  the  integrand  of  (4.32)  and  to  cases  where  more  general  convex  cost 
functionals  (4. 32)  are  considered. 

4.4.  A  Leader  Follower  Game  with  Memory 
Consider  the  dynamic  system 


x(t)  *  Ax (t )  +  B-jU(t)  +  B,v(t),  x(t^)  =  x^,  t€[to,tf] 


o'  o 


(4.43) 


and  the  cost  functionals 
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J1  “  2  F1x(tf)+J'  (*'  (tOQjxCt)  +u'  +v(t)  R12v(t))dt] 

tQ  (4.44) 

1  i 

J-  =  r  [x'(tf)  F„x(t,)+J  (x'  (t)Q  „x(t)  +  u'  (t)R  ,.u(t)  +v(t)  'R  v(t))dt] 

tQ  Z  (4.45) 

where  the  matrices  A,  B.  ,  Q.=Q.'>0,  R.  ,=R.'  .^0  are  piecewise  continuous  functions 

ill  ij  ij 

of  time  over  [tQ,t^]  and  R^,  1*22’  R12  are  nonsingular  Vt€[t  ,t ].  The 
matrices  F^=F^>0  and  the  time  interval  [t^jt^]  are  fixed. 

Consider  the  Stackelberg  game  associated  with  (4.43)-(4.45) .  The 
admissible  strategies  of  the  leader  are  of  the  form 

t 

u(x  ,t)  =J  [  d  (t  ,s )]  x  (s)  (4.46) 

t 

o 

where  T|  is  as  in  (4 . 35)-(4 . 36) ,  so  that  u(*,t)  is  a  continuous  linear  functional 
ml 

on  C ([ tQ , t] ,R  )  for  each  t€[tQ,tj].  The  admissible  strategies  of  the 
follower  are  of  the  form  v(x,t),  x€Rn,  t£R,  where  v  is  continuously 
differentiable  in  x  and  piecewise  continuous  in  t.  All  the  matrices  in 
(4 .43)-(4.45)  are  considered  to  be  of  appropriate  dimensions.  By  x^,  u,  v 
we  mean 

xt  :[to,t]-Rn5  xt(9)  =  x(9),  V8  ,t€[to,tf]  (4.47) 

iT(t)  =  u(xt,t),  v(t)  =  v(x(t),t)  (4.48) 

where  x(t)  is  the  trajectory  of  (4.43)  for  given  u  and  v.  For  each  choice  of 
u  and  v  the  behavior  of  the  dynamic  system  (4.43)  and  the  calues  of  are 

unambiguously  defined,  assuming  that  the  solution  of  (4.43)  exists  over 
[tQ,t^].  Actually  when  the  strategy  (4.46)  is  considered  one  might  w.l.o.g. 
restrict  T,  to  be  0  for  s>t,  t€[to,t^]  .  The  costs  of  the  leader  (J^)  and 
of  the  follower  (J2)  are  functions  of  u  and  v.  We  denote  by  U  and  V  the 
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sets  of  the  admissible  strategies  for  the  leader  and  follower  respectively. 
With  these  explanations  the  LF  game  associated  with  (4.43)-(4.45)  is  clearly 
defined. 

In  the  sequel  we  single  out  a  subclass  of  LF  games  with  the  nice 
property  that  the  leader  achieves  the  best  possible  outcome  for  himself;  i.e., 
the  leader's  and  follower's  strategies  constitute  together  an  optimal  control 
law  for  the  control  problem  with  cost  functional  J^(u,v)  subject  to  the 

constraints  of  the  state  equation.  The  procedure  followed  is  the 
following:  First  solve  the  leader's  problem  as  a  control  problem  with 

controls  u,v.  Let  (u*(t)  ,v*(t))  ,x*(t)  be  the  optimum  control  pair  and 
trajectory,  where  u*(t),v*(t)  are  piecewise  continuous  functions  of  time. 
Consider  any  function  u€  U  such  that  u(x*,  t)  =  u* (t)  T/tC  [  tQ ,tf]  .  Second, 
solve  the  following  inverse  control  problem:  with  u  =  u  in  the  follower's 
cost,  and  the  state  equation,  minimize  J2(u,v)  and  seek  conditions 
so  that  v*  solves  this  problem  and  the  resulting  optimal 
trajectory  for  this  problem  is  again  x*(t).  So,  if  these  conditions  are 
assumed  to  hold  apriori,  then  the  pair  (u,v*)  constitutes  a  Stackelberg 
pair.  One  may  derive  conditions,  by  solving  the  inverse  control  problem, 
where  u  depends  only  on  x(t),  or  on  almost  any  subset  of  {x(T);tQ<T  <  t} 
for  each  t.  One  may  also  single  out  a  whole  class  of  LF  problems 
where  the  inverse  control  problem  does  not  have  v*  as  its  solution,  whatever 

tf_i 

is  the  u.  For  example  if  =J  v(t)  v(t)dt,  then  v*  will  be  optimum  if  and 

co 

only  if  v*(t)  20.  It  is  trivial  to  exhibit  now  a  class  of  J^s  and  A,  B^ , 

B2,  so  that  v*(t)  ^0. 

Consider  the  control  problem 


94 


minimize 

subject  to  u ,  v  piecewise  continuous  functions  of  t 
and  (4.43) 

Then  (4.49)  has  the  solution 


(4.49) 


u*(t)  =  -R^B|Kx(t),v*(t)  =  -R^Kx^) 


-1. 


where  K  is  the  continuous  solution  of 

-1_, _ -1. 


-K  *  KA+A,K+Q1-K[B1R'1B| +B2R^B2]K,K(tf)  =Fl>  t6[tQ,tf] 


(4.50) 


(4.51) 


which  is  assumed  to  exist.  Let  §(t,t  )  be  the  transition  matrix  of  the 
resulting  closed  loop  system  in  (4.43),  i.e. 

3*<t,0  ,  . 

- ^ -  -  (A-B^^K-B^B’K)#^,^),  l(to,to)=I,  t€[t0,tf].(4.52) 

Then  the  optimal  trajectory  x*  and  control  values  of  u*,v*  for  C4.49)  are 
given  by 


x*(t;to,xo)  -  *(t,t0)x0 
u*(t)  =  -R^B{Kf  (t,tQ)x0 
v*(t)  -  -Rj2B^K$(t,to)xo. 


(4.53) 

(4.54) 

(4.55) 


Let  T)  be  as  in  (4.35),  (4.36)  with  n(t,0)  =  C  for  6>_t  and  satisfy 
the  identity 

J  Uslri(t,s)]$(s,t)  =  -R^J(t)B][(t)K(t),  t€[tQ,tf].  (4.56) 

o 

If  T)  satisfies  (4.56),  then 


t  t 

u*(t)  =  J  [dsTl(t,s)]f  (s.t  )x  =J  [d  rt(t>s)]x*(s), 

t  t  5 

o  o 


(4.57) 
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_  ! 

(4.56)  characterizes  all  the  n's  which  result  in  the  same  u*(t)  (4.54),  i.e.  i 

provides  a  class  of  different  representations  of  u*(t)  as  a  linear  continuous 

functional  of  x*  =  {x*(9);tQ<9  <  t}  -  This  class  of  T]'s  is  not  empty,  since 

for  example  j 

fO  ,  for  9  > t,  t€ (tQ,tf] 

-R“J(t)Bj(t)K(t),  for  0<t,  t€(to,tf]and  (4.58) 

for  9<t0,  t-tQ 

satisfies  (4.56)  .  For  fixed  t,  the  set  of  all  Tl(t,*)  which  satisfy  (4.56)  is 

m^xn 

the  hyperplane  Ht  =  D)  (t ,  •  )  1 11  (t ,  •  )  €  NBV([  tQ ,  t]  ,R  ),  T)(t ,  •)  perpendicular  to 
5(.,t)}  shifted  by  71(t,.)  from  the  origin  in  the  dual  space  of  C([t  ,t],Rnxn). 

A  useful  class  of  T]'s  which  satisfy  (4.56)  is  given  by 


Tl(t,s)  -  T|(t,s)  +Ho(t,s)  +.1^.(6, s)d(s-p.(t))  (4.59) 

nuxn 

where  Hq  is  absolutely  continuous  in  s  for  each  t,  :  [t  ,tf]  X  R-R  1 
continuous,  Pi:[to,tf]-R  continuous,  d(s)=0  for  s<0  and  d(s)=l  for 
s  >  0 ,  and 


t  SHo(t,s)  P 

J  - ai -  *(s,t)d8+iEiA1(t,pi(t))*(Pi(t),t)  =  0  on  [to,tf].  (4.60) 

o 

Another  T]  which  satisfies  (4.56)  is 


0,  for  e  >  t/2,  te(tQ,tf] 


T>(t,s)  =/ 


0,  for  9  >  t  ,  t  =  t 
o  o 

-1 


■R‘1(t)B{(t)K(t)J’  $  (t,a)dcr  •  , 

0 


(4.61) 


for  9<rt/2,  t€(tQ,tf] 


N  -Ri"1i(to)B;<to)K(to>>  f°r  e  <  v 


t  =  t  . 
o 
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Notice  that 

t-t 

\+~r 

B*CtJ-j*  [d  J(t,a)]x*(a) 

t 

0 

i.e.  only  the  first  half  of  the  trajectory  up  to  time  t  is  used  in 
calculating  u*(t). 

Theorem  4,2 :  Assume  that  there  exists  a  function  11*  as  in  (4.35), (4 

11*(t,9)=0  for  9  >  t  and  an  nxn  matrix  function  P  :  [t  ,tj  -  R1^11  which 

o’  f 

J  [dgT)*(t,s)]f  (s.t)  =  -R^(t)B^(t)K(t),  t€[to,tf] 

tO 

R22(t)B2(t)P(t)  =  R^(t)B^(t)K(t),  t€[to>tf] 
tf 

P(t)  +J  {-A'  (t)P(t)-Q,  (t)  +T1*'  (T,t)B.*  (t)P(t) 

t 

+  M*'  (T,t)R21(T)R^(T)B][  (t)K(t)}$  (T,t)dr  =  F2-  (tf>t)>  t€[tQ,tf 
Then  the  pair 

t 

u*(x  ,  t)  =  J  [d  Tl*(t,s)]x(s) 

C  t  3 

o 

v*(x(t),t)  =  -R-lB|(t)K(t)x(t) 

constitutes  an  equilibrium  pair  for  the  LF  game  associated  with  (4.43) 
for  any  xq  with  strategy  spaces  U  and  V. 

Proof;  We  set 

Ht)  -  P(t)«(t,tQ)x0. 

Then  the  vector  X  (t)  =  (-l,\(t)')'  and  the  control 

v  *  -R^B^<t)  • 


(4.62) 

.36)  with 
satisfy 

(4.63) 

(4.64) 

]  (4.65) 

(4.66) 

(4.67) 
-(4.45) 

(4.68) 

(4.69) 


satisfy  the  sufficiency  conditions  of  Theorem  4.1  for  the  problem 


minimize  (a* (x£ , t ) ,  v) 


(4.70) 


subject  to  v€V  and  (4.43) 


where  u  is  kept  fixed  equal  to  u*.  That  the  u*  in  (4.66)  is  the  leader's  best 

reaction  to  v*  in  (4.47)  is  an  immediate  consequence  of  the  fact  that  the  pair 

(4.66),  (4.67)  solves  the  problem  (4.49).  q 

The  case  where  the  leader's  strategy  is  allowed  to  be  of  the  form 
t 

J  [  d  T)  (t ,  s )]  y  (t , s ) 

t 

o 

where  y (t ,x)  =  C(t ,x)x (s),  -p-  C(t ,s)  =  0,  a.e.  t  <s  <  t  <  tf  with  T!  (t , s)  •  C(t , s)  as 

as  or 

in  (4.35)-(4.36)can  also  be  considered.  The  property  j-C(t,s)=0,  a.e. 

t  t s 

t  £s  <t<tf  allows  one  to  write  f  [d  T|  (t,s)]y(t,s)  *  f  d  [  T)  (t,s)C(t,s)]x(s) 

O  I  H  S  V.  s 

Co  tO 

and  thus  to  use  directly  Theorem  4.1.  We  only  mention  that  in  this  case 
the  leader  has  restricted  memory  and  7]*-C  should  play  the  role  of  T*  in 
(4.63)-(4.67)  in  the  corresponding  sufficiency  conditions. 

For  given  T]*,  (4.65)  is  an  integral  equation  for  P(t) .  Since  it 
has  a  Volterra  kernel,  if  in  addition  it  holds  that  A'  (T)-71*'  (T,t)  'b^T) 
is  bounded  by  some  M  for  any  t^s  TSt,  tQstst^,  then  the  Neumann  series 
for  (4.65)  is  always  uniformly  convergent  and  furnishes  the  unique  solution 

of  (4.65),  see  [48] .  If  Ti*(t,s)  is  of  the  form 
k  i  .  pxm, 

Tl*(t,s)  =  (s),  H.(t)  6  R  \  H1  (s)  €  Rnxp  (4.71) 


then  (4.65)  can  be  written  as 
tf  . 

P(t)+J  {-A'(t)+ZH  (t)H1(T)Bj(T)}P(T)«(T,t)dT-F2l(tf,t)  + 

*;f  1 

+J  [Q2(T)-n*(T,t)R21(T)R1J(T)B{(T)K(T)}f  (T,t)dT  (4.72) 

which  is  an  integral  equation  for  P  with  a  kernel  of  finite  rank  and  thus 
its  solution  is  of  the  form 
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P(t)  *M(ta,t)+SHl(t)Ei(t  ,t)  (4.73) 

where  ]_>•••  are  constant  matrices  which  can  be  found  as  solutions  of 
algebraic  linear  equations.  In  this  case,  checking  (4.64)  is  easy  as  soon 
as  the  Hi's  in  (4.73)  are  found. 

If  (B^KB,,)  1  exists  over  [tQjtf3,  (it  suffices  that  rank  B2  =  m2  and 
F^>0),  then  (4.64)  is  equivalent  to 

P(t)  =  M(t)  +Y (t)  ,  M(t)  =  KB2(B^KB2)‘1R22Rj2B^K  (4.74) 

B2 (t)Y (t)  so  on  [te,tf]  (4.75) 

and  (4.65)  can  be  transformed  into  an  integral  equation  for  Y. 

Iheorem4.2  suggests  that  for  an  LF  game  with  given 
A,  B^ ,  Qi,  R_,  F.^,  one  may  try  to  find  T]*  and  P  which  satisfy  (4.63)-(4.65) 
and  then  consider  (4.66),  (4.67)  as  a  solution.  Also,  by  solving  (4.65) 
for  Q2>  one  can  exhibit  a  whole  class  of  LF  games  with  solution 
(4.66),  (4.67),  where  Tl*,  P,  K,  A,  B^  B£,  F^  Rn,  R12,  R22 
are  chosen  as  to  satisfy  (4»51),  (4.52),  (4.63),  (4.64),  F2=P(t^)  and  R21 
is  chosen  arbitrarily. 


4.5,  Special  Cases  and  Generalizations 

We  first  apply  the  results  of  Theorem  4.2  to  two  special  cases. 

Case  i:  Let  ^*=11  as  in  (4.58).  Then  u*  in  (4.66)  assumes  the  form 

u*(xfc,t)  =  -R'jB'Kx(t). 

(4.63)  is  satisfied  and  (4.65)  simplifies  to 


If  (B^KB^  1  exists  and  is  differentiable  on  [tQ,tf]  and  b2>  kr22R12B2K  are 
differentiable  on  [tQ,tf]  and  of  constant  rank,  then  all  the  R22,  Q2>  F2, 

P  with  R22  >  0,  P>  0  which  satisfy  (4.64)  and  (4.76)  are  given  by  (see  [43]) 


R22  =  VrV’’ 

(4.77) 

P  =  M  +  Y 

(4.78) 

q2  =  -p-  p(a-b1rJ}b|k)  -  (a-b^JbJid'p 

(4.79) 

-KElRilR12RilBiK  +  PB2R22B2P 

(4.80) 

F2  -  Q2(tf) 

(4.81) 

B2KB2  =  V^V  A  =  Jordan  diagonal  form 

(4.82) 

ta  =  Ar,  r  =  r'  >o 

(4.83) 

B'Y  =0,  Y  =  Y'  S  0. 

(4.84) 

If  r  and  Y  do  not  satisfy  T  >  0,  Y  £  0  then  one  cannot  conclude  that 
R22  ^  0  and  P  s  0  resDectively.  Y  and  R^2  have  to  be  chosen  properly 
differentiable  so  that  P  exists  and  is  piecewise  continuous  The 

above  construction  does  not  guarantee  Q2S0,  F2^0. 

Case  ii:  Let  7]*  =  T)^  +T|2  where 
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pRll  (t^B{  (t)LL  (t), 

T)L  (t,s)  'Rll(to)Bi(to)Ll(to): 

0  , 


/ 


T1  (t,s)  *  \ 


(s-t)L2(t), 

0 


"1 3  2 


for 

s  <  t 

t€(to,tf]. 

for 

S^V 

for 

s  kt, 

t€(tQ,t  ]  and 

for 

s  >  t  , 
0 

t  =  t 

0 

for 

s  <  t, 

t€[to,tf] 

(4.85) 

for 

s  >  t 

Then  u 

*  in  (4. 

.  66)  assumes  the 

form 

t 

u*(xt,t)  =  -R11(t)B|(t)L1(t)x(t)  +L2(t)  J  x(s)ds 

Co 

and  (4.63),  (4.64),  (4.65)  simplify  to 


(4.86) 


(t)B|(f)L1(t) +L2(t)J  $(s,t)ds  =  -R'J(t)Bx(t)K(t) 

tO 


R22(t)B^(t)P(t)  =  -R^(t)B^(t)K(t) 

Cf  , 

P(t)  +  J  f-A’  (t)P(t)-Q2(t)-L^(t)B1(t)r11(t)B{(t)P(t)  + 


(4.87) 

(4.88) 


-Kt-T)L^(T)B{(T)P(T)-L[(T)B1(T)R^(T)R21(T)R^(T)B|(T)K(T)  + 

-Kt-r)L^(T)R21(T)R^(T)B{(T)K(T)}$(T,t)dT  =  F2«(tf,t).  (4.89) 

The  cases  i  and  ii  were  special  cases  of  the  case  considered  in 
the  previous  section.  We  will  consider  now  cases  where  the  leader  uses  the 
previous  strategy  values  as  well.  In  the  LF  game  considered  in  Sec¬ 
tion  4.4  ,  the  value  of  the  leader's  strategy  at  time  t  was  allowed  to  depend 
on  the  previous  trajectory  x  »{x(9);  t  i9st].  More  generally  one 

may  allow  that  the  values  u(t)  of  the  admissible  strategies  of  u  of  the  leader 
at  time  t  depend  not  only  on  the  previous  values  of  x  but  also  on  those 

of  v.  Assuming  this  dependence  to  be  linear  we  have 
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c  c 

u(t)  =  J  [dgTl1(t,s)]x(s) +J  tds^  (t,s)]  v(s) 


or  more  generally 

t  t  _  t 

q  (t )  =  J  EdgTl1(t,s)]x(s)+J  [dgTl2(t,s)]u(s)+J  [dsTl3(t,s)]  v(s)  (4.90) 

C0  Co  Co 

qSL^fixed.4  (The  T|  1  ,T12 ,T]3  in  (4.90)  are  as  in  (4.35),  (4.36).)  So,  foragiven 

choice  *"^3  C^e  ■'■eader>  follower  is  faced  with  the  problem 

1  *f 

minimize  j  E x ' (tf )F2x (tf )  +J*  (x' (t)Q2x (t)-KT' (t)R2]u(t)4v(t)R22v(t))dt] 

Co 

subject  to  x (t )  =  Ax(t)  +  B^u^t)  +B2v(t),  x(tQ)=xo  (4.91) 

(4.90)  and  u,v  piecewise  continuous  functions  of  time. 

Theorem  4.1  can  be  now  used  to  derive  sufficeint  conditions  for  problem  (4. 91) . 

A  simple  version  of  (4.90)  is 

t  t  _ 

u(t)  -J*  tds~1(t,s)]x(s)  +  J*  (d^2(t,s)]z(s)  +  L(t)v(t) 


Co  to  (4.92) 


where 


z(t)  =  AL(t)x(t)  +A2(t)z(t) +B1(t)u(t) +B2(t)v(t),  z(to)  =  z0, 


(4.931 


and  the  matrices  L,  A^,  are  real  valued  piecewise  continuous  functions  of 

l 

time  and  z(t)€R  ,  l  arbitrary.  For  the  linear  system  (4.43)  with  quadratic 
costs  (4.49) , (4.45) ,  we  augment  (4.93)  to  (4.43),  set  x  =  (x'z') 1  and  the 
system  is 


Notice  that  in  (4.90),  u(t)  depends  on  its  own  previous  values .  If 

u(t)  were  allowed  to  be  any  function  of  x(0),V(0),  t  S0st,  then  the 
dependence  of  u(t)  on  it's  previous  values  would  not°buy  the  leader  anything 
additional.  But  if  u(t)  is  restricted  to  deoend  on  x(9),  v(0),  t  s Q.s  t  in 
a  special  form  (like  in  (4.90),  seealso  (4.92)  -(4.96)),  then  allowing  dependence 
of  u(t)  on  it's  own  previous  values  will  benefit  the  leader. 


J 
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A 

0 

B-j 

Bo 

X 

x(t)  = 

Ji 

^2 

x(t)  + 

1 

,\ 

u(t)  + 

2 

_Y 

v(t),  x(tQ)  = 

0 

z 

0 

=  Ax  +  B^u  +  I^v, 


with  costs  J.,J2  as  in  (4.44),  (4.45)  and  with  the  strategy  of  the  leader 
restricted  to  be  of  the  form 

t 

u(x  , t)  =  J  [d  Tj(t,s)]x(s)  +  L(t)v(t) .  (4.95) 

t 

o 

The  results  of  Section  4  are  directly  applicable  to  (4. 94)- (4.95)  and  the 
problem  is  to  find  Tj ,  ^ L,  P  so  that  (4. 63)  -  (4. 65)  are  satisfied  wherein 
(4. 63)-(4. 65)  one  should  use  AjB^,  (B2+B^L)  in  place  of  AjB-^B,,.  As  far  as 
it  concerns  zq,  it  may  be  set  arbitrarily  equal  to  a  constant  or  to  a 
function  of  xq  preferably  linear.  The  choice  of  zq  might  affect  not  only 
the  feasibility  of  (4. 63) - (4. 65)  but  the  follower's  optimum  cost  value  as 
well.  A  simpler  case  of  (4.95)  is 

u(t)  =  L^x(t)  +  L2z(t)  +  Lv  (t)  (4.96) 

in  which  case  the  solution  of  the  LF  game  is  easy  since  the 
leader's  controls  are  actually  A^ ,  B^,  B2,  L^,  L2 ,  L,  i.e.  the  leader 

plays  open  loop.  Nonetheless  the  leader's  problem  will  be  nonlinear  since 
his  control  multiplies  the  state  (tf, z1)'. 

4.6.  A  Nash  Game  with  Memory 

Consider  the  Nash  game  associated  with  (4.43) - (4.45)  where  at  each 
instant  of  time  t,  both  players  have  access  to  all  the  previous  values  of 
the  state.  The  admissible  strategies  for  both  players  are  of  the  form 


UJ 
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t 

u(xt,t)  *  J  [dgTl1(t)s)]x(s) +b1(t)  (4.97) 

Co 

t 

v(xt,t)  *  J*  [dsTl2(t,s)]x(s) +b2(t).  (4.98) 

Co 

"H ^  and  u2  are  as  in  (4.35)-(4.36),  b^(t)  are  piecewise  continuous  functions  of 
time  with  appropriate  dimensions.  By  x£  ,  u,  v,  we  mean 

u(t)=u(xt,t),  v(t)-v(xt,t)  (4.99) 

xt:  [to,t]-Hn,  xt(8),  Ye€[to,t],  Yt€[to,tf] 

In  the  next  Proposition  we  give  sufficient  conditions  for  a  pair 
of  the  form  (4.97),  (4.98)  to  constitute  a  Nash  equilibrium  pair.  The  first  part 
of  the  Proposition  refers  to  a  particular  initial  point  xq,  while  the 
second  part  gives  conditions  similar  to  the  coupled  Riccati  differential 
equations,  see  [  1  ]  ,  which  result  in  solutions  in  feedback  form  which  are 
solutions  for  any  initial  point  x^. 

Proposition 4. 1:  (i)  Assume  that  there  exist  T|*,  T]*  as  in  (4.35)-(4.36),b*,  b* 

piecewise  continuous  and  n,,  .  [t  ,  t  of  bounded  variation  which 

r  1  2  o  f 

satisfy : 


+Rij(T)Rjj(T)Bj(T)P'j(T)]dT  =  Fix(tf)’  (4.100) 

b*(t)+J  ldgTlJ(t,s)]x(s)  =  -R'}(t)B^(t)p.i(t),i=l,2  (4.101) 

fco 

x (t )  =  A(t)x (t)  -B1(t)RjjB|(t)^1(t)-B2(t)R^2(t)B^(t)n2(t) 

*(t0>  "V 


(4.102) 
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Then  the  strategies 

t 

u*(xt,t)  =  J  [dsD*(t,s)]x(s)+b*(t)  (4.103) 

Co 

t 

v*(xt,t)  =■  [dsT)*(t,s)Jx(s)+b*(t)  (4.104) 

Co 

constitute  an  equilibrium  pair  for  the  Nash  game  associated  with  (4.43) - (4.45) 
with  admissible  strategies  (4.97),  (4.98)  and  with  x(to)=xQ. 

(ii)  Assume  that  there  exist  as  in  (4.35)-(4.36)  and  matrix  functions 

P^,P2:  [tQ,t^]  -*RnxRn  bounded  variation  which  satisfy 


<'T)8j(f)p(T)])$(r,t)dT=' 


-Pi«(tf,t)  i.J-1,2,  i*j  (4.105) 


, 

3t  =fArt>-Bi(t)RU(c)B’(t)P1(t)-B2(t)R22(t)B^(t)P2(t)]J(t,t0)  •  (^- 106) 


*(tO»to)“I 


(t)B^(t)Pt(t),i-l,2  • 


(4.107) 


Then  the  strategies 


u*(xt,t)  =  J*  [dg71f(t,a)]x(s) 
t0 

nc 

v*(*t,t)  -  J  [dgT,*(t,s)]x(s) 
l0 


(4.108) 


(4.109) 


constitute  an  equilibrium  pair  for  the  Nash  game  associated  with  (4.43)  - 
(4.45)  with  admissible  strategies  (4.97),  (4.98)  and  for  any  xQ€Rn. 
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Proof :  (i)  If  the  second  player  plays  (4.98)  then  (4.100)  With  i=l,j=2  and 

u(t)  =  -R"J(t)B^(t)lx1(t)  (4.110) 

constitute  sufficient  conditions  for  optimality,  by  Theorem 4.1,  of  u  for 
the  control  problem  faced  by  the  first  player.  (In(4.100)  the  term  **22^2^2 
is  substituted  by  -J*  [d  Tl^lx  in  these  sufficient  conditions.)  Similar 

f-  s  z 
to 

reasoning  applies  for  the  control  problem  faced  by  the  second  player  when 
the  first  player  plays  (4.103). 

(ii)  We  will  first  seek  solutions  |i,  p.„  of  (4.100)  which  will  work  for  any  x  . 

i.  >  z  o  • 

Let 

(J-i(t)  =  P].(t)f  (t,to)xQ  (4.111) 


where  $  as  in  (4.106).  Using  (4. 109)  in  (4. 100)  and  (4. 101)  we  obtain  (4. 105)  and 
(4.107)  where  we  considered  bi  s0.  It  is  clear  now  that  if  (4.105)  and  (4.107) 
hold,  the  H,i's  as  in  (4.107)  satisfy  (4. 100) - (4. 102)  .  O 

The  case  where  the  players  use  strategies  of  the  form 


u(ylt,t)  ■  J*  [ds1l1(t,s)]y1(t,s)  +b1(t) 


v(y2t,t)«  /  fdsTl7(t,s)]y7(t,s)  +b?(t) 


s  2 


where  for  i  ■  1,2 : 


y^t.s)  =  Ci(t,s)x(s),  tQ<  s  <  t, 

3“  ^(t.s)  -  0  a.e.  tQ<s<t<tf, 
s 

n. 

yi(t,s)  ?R  1 


(4.112) 
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and  7]  (t,s)  *T|  (t,s)C^(t,s)  are  as  in  (4.35)- (4.36)  can  also  be  considered. 

The  strategies  (4. 112)  correspond  Co  the  case  where  the  i-th  player's  informa¬ 
tion  at  time  t  is  {<h  (t , s )x (s ) ;  t  <s<t1.  We  only  mention  that  in  this 
case  Ti"Ci  should  play  the  role  of  T)*  in  the  conditions  of  Proposition  4. 1 

The  results  of  Proposition  4. 1  (see  also  problem  (P ' ) )  can  be  used 
to  study  the  Nash  game  associated  with  (4.43)  -  (4.45)  where  the  players  use 
previous  values  of  their  opponents  strategy  values.  For  example 

u(t)  =  J  [dgTl11(t,s)]x(s)  +  J  [  dgT!  12  (t,s)]  v(s) 

to  to 

__  t  nt 

v(t)=J  [  dgTl21  (t  ,s)]x(s)  +  J  [  ds'H22  (t ,  s )  ]  u(s) . 

Co  to 

Strategies  of  the  form  (4.92),  (4.95),  (4.96)  can  be  considered  for  the  Nash  game 
and  the  augmentation  (4. 93) - (4 . 94)  may  also  be  employed  in  this  case.  The 
procedure  for  studying  sufficiency  conditions  for  Nash  games  with  such 
strategies  should  be  obvious  by  now  and  we  will  not  take  it  up  here. 


j 


u 


107 


CHAPTER  5 

ON  SOME  STOCHASTIC  STATIC  AND  DYNAMIC  NASH  GAMES 

5.1.  Introduction 

In  this  chapter  we  deal  with  stochastic  Nash  games.  We  start 
in  Section  5.2  by  considering  a  static  stochastic  Nash  game,  where  each 
player  has  a  quadratic  cost  and  his  information  is  a  linear  function  of  a 
Gaussian  random  variable.  Certain  known  results  are  considered  first  and 
some  new  ones  are  provided  concerning  the  existence  and  uniqueness  of  the 
solution.  In  Section  5.3  we  study  solutions  of  the  game  of  Section  5.2  which 
are  affine  in  the  information  and  provide  a  method  for  finding  them.  In 
Section  5.4  we  generalize  some  of  the  results  of  the  previous  sections  to 
the  case  where  each  player's  control  vector  is  subdivided  into  smaller 
control  vectors,  each  one  of  which  has  to  use  different  information.  The 
information  available  to  the  subvectors  of  each  control  vector  is  nested. 

In  Section  5.5  we  consider  a  discrete  time  stochastic  Nash  game  with  linear 
stochastic  state  equation  and  quadratic  costs  where  the  players  have  noise 
corrupted  state  measurements.  Special  cases  of  this  game  were  solved  in 
[17]  and  [63].  The  case  where  both  players  have  perfect  state  measurements, 
i.e.  C.  ,  ■  I  and  v.  ,  =  0  Vi,k  (see  (5.51)  -  (5.55))  was  studied  in  [17]  and  it 

1  )K  1  f  1C 

was  shown  that  if  the  noise  w^  in  the  state  equation  is  nondegenerate,  then 
the  game  admits  a  unique  solution  affine  in  the  information  under  inver- 
tibility  conditions  for  certain  matrices.  It  was  also  shown  in  [17] 
that  if  w  is  degenerate,  then  the  game  will  have  in  general  an 

K 

infinite  number  of  nonlinear  solutions.  The  case  where  at  each  stage  k 
the  players  share  their  previous  state  measurements  and  their  information 
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differ  only  in  the  k-th  state  measurement  was  studied  in  [63]  where  it  was 
shown  that  the  game  will  admit  a  unique  solution  affine  in  the  information 
under  invertibility  conditions  for  certain  matrices.  In  the  more  general 
case  where  the  assumptions  of  [17],  [63]  concerning  the  information  of  the 
players  do  not  hold  the  solutions  of  the  game  becomes  extremely  difficult. 

In  Section  5.5  we  single  out  some  new  classes  of  problems  which  can  be 
relatively  easily  solved  by  using  the  results  of  the  previous  sections.  In 
the  last  Section  5.6  we  translate  some  of  the  results  of  Section  5.5  to  a 
continuous  time  stochastic  Nash  game  with  a  linear  stochastic  state  equation 
and  quadratic  costs  where  the  players  have  noise  corrupted  state  measurements. 
Examples  demonstrating  certain  properties  of  the  solutions  are  also 
given. 


5.2.  A  Stochastic  Static  Nash  Game 

In  this  section  we  consider  a  static  stochastic  Nash  game.  After 
a  brief  review  of  some  results  already  available  in  the  literature  we 
examine  certain  properties  of  this  game  and  present  some  new  results. 

Let  x  be  a  Gaussian  random  variable  over  a  probability  space 
(ft,<J,p),  x  :  ft -*■  Rn  with  mean  x  and  covariance  E,  and  tw0  random 

variables  defined  by 

yt  -  HiX,  i-1,2  (5.1) 

where  is  an  n^  * n  real  constant  matrix.  Consider  the  Nash  game  with  two 

players,  1  and  2,  where  player  i  chooses  a  Borel  measurable  function 
ni  mi 

u^  :  R  -*•  R  and  his  cost  is 
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J i (ui * u2>  =  Ui (yi)ui (yi> +  ui (yi)Qix + ui (y*) Vj  (yj ) +  u  • (yi)hi ] 

i#  j,  i,j-l,2.  (5.2) 


Qi»  R^,  are  real  constant  matrices  of  appropriate  dimensions,  is 
referred  to  as  the  information  available  to  player  i.  We  also  want  u^,u2 
to  be  chosen  so  that  u^(y^)  ^2^2}  have  finite  second  moments.  Thus  our 
problem  is  to  find  (u^.u^)  which  satisfies 


Jl(u*,u*)  <  J^Cu^.u*),  V  admissible  u^ 
J2(u*,u*)  <  J2(u*»u2) ,  V  admissible  u2> 


(5.3) 


A  straightforward  application  of  Radner's  theorem  [51],  [52]  results  in  the 
following  theorem  (Theorem  1  of  [52]). 

Theorem  5.1:  (u*,u*)  is  a  Nash  equilibrium  pair  for  the  game  defined  above 

if  and  only  if 


Ul^yl^  =  -EtQix+Rxu2^2^ +hl^yl-®  (5-4) 

u2(y2)  =  -E[Q2x+R2u1(y1)+h2|y2].  (5.5) 

Substituting  u2  from  (5.5)  into  (5.4)  we  obtain 

Ul(yl)  "  -Q^txlyxl-hj^  +  Rxh2  +  R1Q2E [E [x | y2 ]  |yil 

+  R1R2E[E[u1(y1)  |y2]  lyj^] .  (5.6) 

Therefore  the  investigation  of  solutions  of  this  Nash  game  is  equivalent  to 

the  investigation  of  solutions  u^  of  (5.6).  It  can  be  easily  shown  that 

since  the  operator  P^*E[*|y^]  is  a  projection  operator  in  an  appropriately 

defined  Banach  space  L  ([52  ]  ,[53.]) ,  it  has  norm  <1,  and  thus  the  operator 

P  -  R1R2E[E[- [  y2  ]  |y1]  has  norm  |jp[|<  IjR^II-  Thus  if  II  Pit  CIlR^II  <  1,  the 
-1  2 

expansion  (I-P)  *I+P+P  +•••  holds  ([54],  p.  231)  and  thus  (5.6)  has 


a  unique  solution  given  by 


no 


uL( y x)  =  (I  +  P  +  P  +  ••■)(-Q1P1x  +  R1Q2P1P2x-h1  +  R1h2). 


(5.7) 


Since  and  are  given  by  (5.1)  and  x  is  Gaussian  we  conclude  that  the 
converging  infinite  sum  (5.7)  is  a  sum  of  Gaussian  random  variables  each 
one  of  which  is  affine  in  y^  and  thus  we  conclude  that  u^(y^)  will  be  of 


the  form 


Ul(yl)  =  Llyl  +  dl 


(5.8) 


where  L^,d^  are  some  constant  real  matrices.  If  (5.8)  holds  then  we  see 
from  (5.5)  that  u2(y2)  will  be  of  the  same  form,  i.e. 

u2(y2 )  *  L2y2  +  d2'  (5’9) 

The  above  discussion  is  formalized  in  the  following  theorem  of  [52]. 

Theorem  5.2:  If  II  R^R^I  <  1  or  li  <  1  >  then  the  Nash  game  has  a  unique 

solution  which  is  of  the  form  (5. 8),  (5. 9). 

Theorem  5.2  guarantees  existence  and  uniqueness  of  a  Nash  solution 
and  that  the  optimal  strategies  will  be  affine  in  the  information.  It  is 
crucial  to  notice  that  the  condition  II <1  has  nothing  to  do  with  the 
H^jH2  matrices  which  nonetheless  constitute  a  quite  substantial  part  of  the 
data  of  the  problem.  For  example  if  H^,H2  are  such  that  P^P2*0,  i.e.  P^,P2 
project  into  orthogonal  subspaces  of  L  then  P=0  and  the  solution  u^  of  (5.6) 
is  given  by  the  first  two  terms  of  (5.6)  and  is  of  the  form  (5.8), 
whatever  R^R2  be*  More  generally  one  can  see  that  the  solution  of 

(5.6)  is  unique  if  and  only  if  the  operator  I-R^R2P^P2  is  invertible. 

Theorem  5.3:  Assume  that  P^P2  or  P^  is  a  projection.  Then  if  det (I-R^R^^O , 
the  Nash  game  admits  a  unique  solution  pair  (u*,u2)  with  u*  affine  in  y  . 


Ill 


Proof :  Let  =  P  be  a  Projection.  Then  if  (I-R^^Pj^)^  =  0  (I-R^I^**^  = 

Let^=v’1+^2»  P^2  =  0*  Thus  =  R1R2^1  *"^2  =  ®  ^  (I_RlR2^1  =  °‘ 

Since  det(I-R^R2)  J4  0  =><^  =  0.  Thus  <P  =  0  and  I-R.^P^^  is  invertible.  By 
Theorems  1  and  2,  p.  228  of  ref.  [54],  we  conclude  that  (I-R^^Pj^)  ^  is  a 
linear  continuous  operator.  Thus  u*  is  affine  in  and  consequently  u*  is 
affine  in  y^. 

Corollary  5.1:  If  =  C^H^  (or  =  C2H2)  for  some  matrix  (or  C2) >  then 

P1P2  =  P2P1  =  P2’  ‘Vl^lW* 

Proof :  Let  3  ,3*2  be  tbe  minimai  a-fields  in  fi  with  respect  to  which  y^,y2 
are  respectively  measurable.  If  ^2  =  ^1^1  *2  ~  ‘*1  **  E[E[  •  |j^]  Ijj^ 1  = 

E [ E [  -  J 3*2 1 1 ]  =  E [  *  | if 2 1  by  Theorem  6.9,  p.  260  of  [55],  i.e.  P^P2  ~  P2P1  =  P2 
and  P2  is  a  projection.  * 

Notice  that  if  Ir^^R^I  (or  H  R2  R^  ^  ^  1 ) »  the  expansion  (5.7)  holds  while  it 
does  not  have  to  hold  under  the  assumptions  of  Theorem  5.3. 

Let  us  now  assume  that  for  some  nonzero  vector  e  it  holds 
Rj^e =  e  and  that  are  such  that  they  provide  a  piece  of  common 

information  to  both  players.  Let  us  also  assume  without  loss  of  generality, 
that  x^,  the  first  component  of  x  is  available  to  both  players.  Consider 
any  nonzero  Borel  measurable  real  valued  function  Y(x).  Then 
P^e-'i'(x^)  =  P2e*'f(x^)  =  e>li'(x1)  and,  e’¥(x^)  =  R^^e • '{'(x^)  and  e-f(x^)^0.  Thus, 
if  (5.6)  has  a  solution  and  det^-Rj^)  =  0  this  solution  will  not  be  unique. 
We  thus  conclude  that  the  condition  detCl-R^^)  =  0  is  necessary  for  the 
uniqueness  of  the  solution  of  the  Nash  Game,  if  the  two  players  share  a 
common  nontrivial  part  of  their  information. 

We  conclude  this  section  with  an  example  which  demonstrates  that 
the  Nash  game  may  have  no  solution  at  all  or  an  infinite  number  of  solutions. 
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Example  5.1:  Let  x=(x^,x2)',  ’^2  ,U1  ,u2  ta^e  values  in  R  »  x=0, 


hl  =  h2  =  ° 


and 


cov(x)  = 


"I  ° 
0  a. 


ri 

o  1 

ri 

O' 

II 

H 

r 

o 

o 

1 _ 

X, 

y2  = 

o 

l- 

(5.10) 


(5.11) 


Then  (5.6)  assumes  the  form 


(i-r1r2)u1  =  (-ql  +  rlQ2) 


‘1 

L  o  J 


Let 


'0  1' 

’  R2  = 

.2  1. 

'a  y" 

- 

»  q2  - 

-0  6. 

'1  1/2 


1  2 


(5.12) 


where  1=26  -5a.  Then 


l-h*2 


-Qx  +  RxQ2  - 


-2  0 
-5  0 

'1-a  7-y 

.3-6  11-6 


and  (5.12)  results  in 


w  = 


—  x 
2  X1 

L  ^(x^ 


(5.13) 


where  ^(x^)  is  any  Borel  measurable  function  of  which  results  in  u^(x^) 


having  finite  covariance.  (If  we  had  1  t  2S-5et  then  (5.12)  would  have  no 


solution.)  From  (5.5)  we  obtain 
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u*(x)  = 


^  +  ^(xl)  +  2x2' 
3a- 1 

2  X1  +  7x2 


(5.14) 


and  using  (5.2),  (5.13),  and  (5.14)  we  find 

J*  =  -  \  (a-l)2a1-i  E  ^2(Xl)]  (5.15) 

X1 

2 

I  E  [*2(X;l)  +  2xv>(x1)].  (5.16) 

X1 

Notice  that  by  choosing  ^(x^)  =  nx^  and  letting  n-*-  +  °°  =»  J*  -*■  -  °°  .  Notice 
also  that  the  solution  of  the  game  is  exactly  the  same  even  if  in  the  cost 
functional  of  player  1,  terms  of  the  form  u^x,  x'x  were  included. 

Actually  by  adding  to  the  cost  functional  of  player  1  the  terms  9u2u2, 

y  0x'x,  where  6>0  we  see  that  becomes  a  quadratic  of  the  form 


" 1  Ri  V 

V 

r  t  »  ?  , 

[ul,u2*x  J 

91  0 

u2 

q’  0  01 

X 

L  1  J 

which  is  positive  semidefinite  for  a  sufficiently  large  0  (see  [56]). 
Addition  of  such  terms  to  will  not  alter  the  solution,  but  will  alter 
J*,J2  and  thus  will  eliminate  the  possibility  J*  -  00  .  Finally  notice  that 
^~^1R2  s^n§u^ar  ar,d  =  ^2^2  an<^  t^lus  possibility  of  nonunique 
nonlinear  strategies  was  expected  by  the  discussion  following  Corollary  5.1. 


5.3.  Affine  Solutions  of  the  Stochastic  Static  Nash  Game 

In  the  previous  section  we  stated  conditions  under  which  the 
Nash  game  posed  there  will  have  a  unique  solution  with  the  controls  of  the 
players  being  affine  in  their  information.  The  conditions  det (I-R^R2)  £  0 
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and  demand  some  special  attention.  Let  us  assume  that  H2 = C^H  . 

If  it  happens  that  det (I-R^R^)  *  0 ,  then  by  perturbing  very  little  the 
arguments  of  R^  and  ,  we  can  guarantee  that  for  the  perturbed  pair 
^R1,R2^’  ^  ®  hold.  This  is  important  because  in  a  real  life 

problem  the  values  of  R^,R2  are  aPPr0Ximately  known.  Thus  almost  all 
such  Nash  games  admit  a  unique  solution  which  is  affine  in  the  information. 

On  the  other  hand  if  det(I-R^R2)  =0  but  it  is  very  close  to  zero,  it  might 
happen  that  out  of  the  possibly  infinite  solutions  one  can  result  in  making 
or  J2  arbitrarily  small,  (recall  Example  5.1),  i.e.  if  det(I-R  R^)  =  0 
the  game  might  exhibit  an  unstable  behavior.  Therefore,  in  order  to 
guarantee  uniqueness  of  an  affine  solution  in  a  realistic  framework,  one 
should  impose  a  condition  of  the  form  [ det (I-R^R2) [ >  e > 0 ,  where  e  is  some 
positive  constant  which  depends  on  the  measurement  error  of  the  entries  of 
and  R2<  The  condition  H2 =  can  be  satisfied  if  a  negligible  but 

nonzero  signaling  is  allowed  between  the  players  (see  discussions  in  [57], 
[58],  [59])  and  this  in  conjunction  with  det(I-R^R2)  ^0  will  guarantee 
existence  of  a  unique  solution  affine  in  the  information. 

Let  us  now  restrict  a  priori  our  attention  to  admissible  pairs 
(u^,u2)  of  the  form  (5. 8), (5. 9).  It  is  clear  that  if  player  1  plays  (5.8) 
then  player  2  will  use  a  strategy  of  the  form  (5.9)  and  vice  versa  (see  [51]). 
In  this  case  we  have  the  following  theorem. 

Theorem  5.4:  Assume  that  H2,H2  have  full  rank  and  n2  respectively,  and 

f  t 

£>0,  so  that  H2EH2  are  invertible.  Then,  a  pair  (u^,u2)  of  the  form 

(L^y^  +  dj,,  L2y2  +  d2)  is  a  solution  of  the  Nash  game  posed  in  Section  5.2, 
if  and  only  if  the  following  matrix  equations  are  satisfied. 
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Lx  =  -(C^  +  R^H^EH^EH^)'1  (5.17) 

L2  =  -(Q2  +  R2L1H1)EH2(H2EH2)~1  (5.18) 

dl  =  -W1  +  R1L2H2)(I_  ZH1(H1ZHi)-1hi)5~  (Rld2  +  hl)  (5.19) 

d2  =  -(Q2  +  r2LiHi)(I- ZH2(H2ZH2)~lH2)5‘  (R2dl  +  h2)‘  (5.20) 


Proof:  The  proof  is  a  matter  of  substitution  of  u.“L.y. +d.  in  (5. 4), (5. 5) 

-  l  i  l 

and  identification  of  the  appropriate  matrices.  '  ■ 

Substitution  of  L 2  from  (5.18)  in  (5.17)  results  in 

L1- R1R2L1H1ZH2(H2EH2)“1H2EH:[(HirH;[)"1  =  -Q^EH^ (^EH^)"1 

+  R1Q2EH2(H2EH2)“1H2EH^(H1EH^)“I.  (5.21) 

Since  has  full  rank,  by  introducing  we  can  write  (5.21) 

equivalently  as 

m1-r1r2m1eh2(h2eh2)'1h2eh;[(h1eh^)_1h1  =  -q1eh|(h1eh;[)"1h1 

+  R1Q2EH2(H2EH2)-1H2EH^(H1EH^)"1H1  (5.22) 

Notice  that  u^  =  L^y^  +  =  M^x+d^.  Substitution  of  d2  from  (5.20)  in 

(5.19)  results  in 

di'Wi  =  -(Q1  +  R1L2H2)(I-EH^(H1EHp“1H1)x-h1 

+  R1(Q2  +  R2L1H1)(I-EH2(H2EH2)_1H2)5  +  R1h2.  (5.23) 

Notice  that  as  soon  as  is  found  from  (5.22),  we  obtain  L^  =  M^(H^Hj)  ^ . 

L2  is  determined  from  (5.18)  and  with  L^,L2  given,  the  solution  of  (5.23) 
for  d^  determines  d2  from  (5.20).  Consequently  the  study  of  solutions 
affine  in  the  information  is  reduced  to  the  study  of  solutions  of  (5.22)  ana 
(5.23).  Obviously,  if  det(I-R^R2)  4  0  (5.23)  results  in  a  unique  solution 
for  d.  in  terms  of  M  .  Therefore  we  will  focus  on  (5.22). 
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Lemma  5.1:  If  for  any  pair  (A,y)  of  eigenvalues  of  and 

(H^EH^) respectively,  l#Au,  then  (5.22)  has  a  unique 

solution  in 

Proof The  proof  is  an  application  of  Theorem  1  of  [60].  * 

Notice  that  interchange  of  the  indices  1  and  2  in  the  hypothesis  of  Lemma  5.1 
does  not  result  in  any  benefit  since  for  any  matrices  A,B,  which  commute 
AB  and  BA  have  the  same  eigenvalues. 

Proposition  5.1:  If  det(I-R^R2)  ^  0  and  or  H2=C^H^,  c^en  (5.17)- 

(5.20)  has  a  unique  solution  in  L^.L^jd^.d^. 

Proof:  If  H1  =  C2H2  the  product  EH2 = 

=  IH2(H2ZH2)"1H2EH2C2(H1EH')'1H1=  EH2C’(H1EH^)‘1H1= 
i.e.  it  is  a  projection  and  thus  it  has  eigenvalues  0  or  1.  Since 
det (I-R^R2)  ^  0  we  conclude  that  all  eigenvalues  of  R^R2  are  different  than  1 
and  thus  the  hypothesis  of  Lemma  5.1  holds.  Thus  L^,L2  can  be  found 
uniquely.  Since  det^-R.^)  ^  0  (5.23)  can  be  solved  uniquely  for  d^.  ■ 

Clearly  the  conclusion  of  Proposition  5.1  was  expected  in  view  of 
Theorem  5.3  and  Corollary  5.1.  The  above  results  provide  conditions  for  exis¬ 
tence  of  a  unique  solution  affine  in  the  information.  If  these  conditions  do 
not  hold,  we  might  have  no  affine  solutions  at  all  or  an  infinite  number  of 
affine  solutions.  Notice  that  in  our  results  the  matrices  H^,H2  play  a 
certain  role  while  in  Theorem  5.2  they  do  not. 

From  equation  (5.22)  it  is  obvious  that  the  study  of  solutions 
affine  in  the  information  is  intimately  related  with  the  study  of  the  matrix 
equation 

Y  -  AYB  =  D.  (5. 24) 


In  the  sequel  we  will  deal  in  more  detail  with  this  equation.  Let 

A  -  diag(A, , . . . ,A  ) ,  A  =  diag(B, , . . . ,B„ )  be  the  Jordan  forms  of  A  and  B 
a  1  p  b  i  q 

respectively,  with  A=TAaT  \  B  =  RA^R  We  set  X=T  ^YR,  C=T  ^DR  and  thus 
(5.24)  is  equivalent  to 

X-  AaXAb  =  C.  (5.25) 

We  partition  X  and  C  in  the  form  X  =  (X^  ) ,  C  =  (C^  ),  i = 1, . . . ,p,  j  =  1, . . .  ,q  , 
so  that  (5.25)  is  equivalent  to  the  system  (5.26)  of  the  p-q  independent 
matrix  equations, 


X.  -  A.X.  .B.  =  C.  . ,  i  =  1 . p,  j  =  l,. 

lj  l  l]  ]  lj 


»q 


(5.26) 


We  have 


A.  =  A. I  +  H. ,  B.  =  y.I  +  E. 

i  i  i  J  J  3 


where  H^E^  are  of  the  form 


0  1  0  .  0 

0  0  1  0  ...  0 


0 

Lo 


o  1 

....  0 


with  dimensions  m.xm.,  n  *n.  respectively.  It  holds  H.1  =  0,  E.-*  =  0. 

i  i  j  J  i  3 

Case  1:  1  ■  A^p  .  (5.26)  can  be  written  (we  drop  the  indices  for  convenience) 


X  = 


1-Au 


(C  +  AXE  +  yHX  +  HXE] . 


(5.27) 


Multiplying  (5.27)  by  H,E  we  obtain 


XE  =  rA—  [CE+ AXE2  +  yHXE  +  HXE2] 


HX  = 

HXE  = 


1-Ay 

1 

1-Ay 

1 

1-Ay 


[HC+  AHXE+  yH2X+H2XE] 

[HCE  +  A  HXE2  +  yH2XE  +  H2XE2  ] ' 


(5.28) 

(5.29) 


(5.30) 


We  substitute  HXE  from  (5.30)  into  (5.28)  and  (5.29)  and  then  substitute 
XE,  HX,  HXE  from  (5 . 28) - (5  -  30)  into  (5.27).  We  obtain 


X  =  a.C  +  a  CE  +  a,HC  +  a.HCE  +  ctcXE2  +  a  HXE2  +  a,H2X 
1  2  3  4  5  6  7 

+  ctgH2XE  +  agH2XE2 


(5.31) 


where  are  functions  of  X and  u. •  The  importance  of  (5.31)  is 

2  2 

that  in  the  terms  in  the  right  hand  side  X  is  multiplied  by  E  or  H  . 

Repeating  the  procedure  but  starting  with  (5.31)  instead  of  (5.27),  after  o 

steps  we  obtain  an  expression  of  X  in  terms  of  H^CE  and  HaXET ,  a,x>P. 

If  p  >  min(nr  5n_. )  then  the  terms  H  XE  are  zero  and  thus  after  e  steps  we 

have  obtained  the  unique  solution  of  (5.26).  The  existence  of  a  unique 

solution  could  be  foreseen  by  Theorem  1  of  [60],  since  1/A.p.. 

x  j 

Case  2:  1  =  .  In  this  case  is  nonsingular  and  (5.26)  can  be  written  as 

aTV,  . -X..B.  -  A^C...  (5.32) 

i  ij  ij  j  i  ij 

(5.32)  has  a  solution  if  and  only  if  the  matrices 


'aT1  aT1c. 

V  0  B, 


aT1  0  \ 

i 

0  B. 


3  /  V.  3/ 

are  similar,  [61].  If  similarity  holds  then  (5.32)  has  at  least  one 

solution.  Since  the  homogeneous  equation 

X.  .  -  A  X  B  ,  =  0 
i-l  i  ij  3 


(5.33) 


has  1 =  ,  it  will  have  an  infinite  number  of  solutions  and  so  will  (5.32). 

Methods  for  solution  of  (5.32)  are  known  and  we  will  not  take  up  this  issue 
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5.4.  Generalizations 

In  this  section  we  will  prove  generalizations  of  Theorems  5.2  and 
5.3.  Consider  the  Nash  game  where  player  1  chooses  (u^, . . . ,u^) ,  player  2 
chooses  (v  , . ..,v  ),  their  respective  costs  are 


=  e[t 


N 


E  u!uj  +  I  u!q.x  +  ?  .u.R-.v.+t  E  u!r..u.] 

n  -f  t  j  4=1  v.  .  .  i  4  4  4  J 


2  i-].-! 


=i  i  i 


i=l 


i,J=l  i  ij  j  2 


i.J-1  1 

i/j 


iJ  J 


1  N  N  N  i  N 

=  E[-r  E  v .  v .  +  E  v .  Q .  x  +  E  v . R .  . u .  +  t-  E  v! 
x  2  i=i  i  i  i=i  i  i  i,j=l  ii]  J  2  Uj=i  i 


riiVi] 


(5.34) 


(5.35) 


where  x  is  a  Gaussian  random  variable  and  x.u^.v.  takes  values  in  finite 

i  l 

dimensional  Euclidean  spaces,  u^  is  to  be  chosen  as  a  Borel  measurable 
function  of 


Zi  = 


H.x  +  E  L  u  (Z  ) 
i  j<i  ij  3  J 


(5.36) 


and  v.  is  to  be  chosen  as  a  Borel  measurable  function  of 

l 

z.  *  (y.,  > •  •  •  »y., } .  y,  =  h  x  +  e  l  v  (z.). 

1  1  1  1  i  j<iiJJJ 


(5.37) 


In  addition,  u^,v^  must  have  finite  second  moments.  Notice  that  since  the 

information  structure  of  each  player  is  nested,  see  [62],  we  can  actually 

disregard  the  E  L..u.,  E  L.,v.  terms  in  (5 . 36) , (5 . 37) .  The  matrices 
j<i  il  J  j<i  ij_J 

Qi’^i’Rij  ,Rij  ’  rij  =  rji’  ^ij  =  rji’  Lij ’^ij  ’  are  real»  constant,  of 

appropriate  dimensions. 

Theorem  5.5:  A  pair  ((u*, . . . ,u^) , (v*, . . . ,v*))  constitutes  a  solution  of  the 
Nash  game  associated  with  (5. 34)-(5 . 37)  if 


Ui  -  -E[Qix  +  J  Rijvj+jJirijuj|zi] 

v*  =  -E[Q  x  +  E  R..U. +  E  ?, .v, \z, ]. 
i  i  j  ij  j  j^i  ij  j1  i 


(5.38) 

(5.39) 


Proof :  The  proof  is  in  spirit  the  same  to  the  one  of  Theorem  1  in  [52]. 
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If  we  set 

E[  ‘ 

\z.]  =  P. 

1  l  l 

,  E[ *  | Z±]  = 

P.  and  take  into 
i 

EtvJZj]- 

’  V 

if  i  <  j  , 

we  conclude 

that  (5.38)  and 

f  1 
| 

r  p 
12 

r  p 

1  13  1 

r  p 

IN  1 

;  Rnpi  ri2pi 

i  r 
!  ‘  21 

1  : 

I 

r23P2 

r  P 

2N  2 

• 

1  R21P2  R22P2 

|  PN1 

PN2 

• 

N-l  I 

1  RN1PN  . 

R11P1 


*N1*N 


rinpi 


^NN^N 


r  p 
12  1 


’ii  rv 


RNNPN  |  I  °N 


1 1NP1 


I  r  p  . .  r  P 
'23  2  2N2 


i  N1  1  \J2 


QNPN  x 


(5.40) 


(I-L)a  =  Tx. 


(5.41) 


Since  all  the  projections  P^,P^  have  norm  <1,  one  can  readily  conclude  that 
if  the  matrices  r . . , f , . ,R. . ,R. .  are  chosen  sufficiently  small,  then  the 

ij  ij  i]  ij 

~  _i  '  2 

operator  L  has  norm  <1  and  thus  (I-L)  =I  +  L  +  L  +••*  .  Thus  u  can  be 
expressed  as  a  convergent  infinite  sum  of  Gaussian  random  variables  affine 
in  x  and  thus  u  is  affine  in  x.  We  thus  have  proved  the  following  theorem. 
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Theorem  5.6:  If  the  norms  of  I' .  . ,  T  .  ,  ,R.  .  ,R.  .  are  sufficiently  small,  then 
-  i  j  i  j  ij  13 

the  solution  of  the  Nash  game  ( 5 . 34 )— (5.37)  exists,  is  unique,  and  each 

u.  ,v.  is  affine  in  its  information, 
i  1 

In  the  last  part  of  this  section  we  will  show  how  Theorem  5.3  and 

Corollary  5.1  can  be  generalized  for  the  problem  (5.34)-(5.37) .  In  order 

to  illustrate  the  procedure,  let  us  assume  that  N=2.  Let  P^  =  E[*|z^], 

=  E  [  *  |  ] »  P^XE[*|Z^],  “  E  [  *  J  ]  -  Let  us  also  assume  that  for  some 

matrices  C^,C2  it  holds  H^=C^H^,  H2 =  ^2^1  (recaPP  assumptions  of  Corollary 

5.1).  Also  let  L..=0,  L.  .  =  0  without  loss  of  generality.  Then 
xj  ’  ij  6  y 

^1P1 * Pl^l = P1 ’  P^P2 =  p2Pi = P2 ’  P2P2 = P2^2 = P2  ^see  Theorem  6.5.10,  p.  260 

in  [55]).  (5.38)  and  (5.39)  assume  the  forms 


U1  “Q1P1X  “  ri2?lu2  ~  R11P1V1  “  R12P1v2 

(5.42) 

u2  -  -q2p2x  -  r2lUl  -  r2iP2vi  -  r22P2v2 

(5.43) 

V1  =  '^1P1X  ~  P12P1V2  "  RllPlUl  ”  R12P1u2 

(5.44) 

V2  =  ~^2P2X~  P21Vl  "  R21P2Ul”  R22P2U2' 

(5.45) 

Since  P;LU1  =  U1>  P2U2  =  u2 


^1U1  =  ^1^1U1  =  ^1U1 =  ui*  ^iu2  =  P1P2U2’ 


P2U1  31  P2P1U1  =  Plul  *  ui»  P2U2  =  P2P2U2  =  P2U2  =  u2’  and  since  PlP2  =  Pl*  and 

(5.45)  become 


V1  (I-P12P21)  ("V?12^2)P1X  +  (I  VW  (_P11+ P12P21)ul 

+  (l-p12p21)  (-^12  +  ^12P22)u2 


(5.46) 


u2  =  -Q2p2x  "  ^21U1  "  P22U2  “  ^21(I-I’l2^21)  ('^l"^12^2)PlX 


"P21(I"112^21)  (  Rll  +  ^12R21)ul 

'^21(I"^12^21)  (-R12  +  ^12R22)u2 


(5.47) 
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where  we  assume  that  detd-T^^j.)  ^  0*  Substituting  v^>v2  from  (5.46), 

(5.47)  into  (5.42) ,  (5.43)  we  obtain  a  system  of  the  foim 

u^  =  A^P^x  +  A2Piu2  +  A3u1  (5.48) 

U2  =  B1P2X  +  B2u2  +  B3U1  (5-49) 

where  A.,B.  are  matrices  calculated  after  the  substitution.  If  det(I-B„W  0, 
l  i  2 

we  can  solve  (5.49)  for  u^,  substitute  in  (5.48)  and  obtain 

u^  =  AP^x  +  BP^u^  (5.50) 

where  A,B  are  again  matrices  calculated  after  the  substitution.  If 
det(I-B)?t0,  then  the  operator  I-BP^  is  invertible  and  (5.50)  has  a  unique 
solution  in  u^,  linear  in  y^ .  Therefore  u2>v^»v2  can  be  f°und  uniquely  by 
substituting  u^  in  (5.49)  and  u^,^  in  (5.46) * (5.47) .  We  have  thus  shown 
that  if  certain  matrices  are  invertible,  then  the  game  will  have  a  unique 
solution,  linear  in  the  information  under  the  crucial  hypothesis  that  Z ^  is 
a  linear  transformation  of  Z^.  The  generalization  of  this  procedure  to  the 
case  here  N>2  is  obvious.  We  can  thus  state  the  following  theorem. 

Theorem  5.7:  If  Z„  is  a  linear  transformation  of  Z, ,  or  Z„  is  a  linear 
transformation  of  Z^,  then  if  certain  matrices  are  invertible,  the  game 
( 5 . 34 )— (5.37)  has  a  unique  solution  linear  in  the  information. 

5.5.  A  Discrete-Time  Stochastic  Linear  Quadratic  Nash  Game 

In  this  section  we  consider  a  linear  quadratic  discrete  time, 
stochastic,  Nash  game,  where  the  players  have  noise  corrupted  measurements 
of  the  state.  The  aim  of  this  section  is  to  point  out  several  difficulties 
and  provide  solutions  or  conditions  for  existence  of  solutions  for  some 


special  cases. 
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Consider  the  dynamic  system  whose  state  x^  evolves  in  accordance 
with  the  equation 

Vl  =  Vk  +  Bl,kul,k  +  B2 ,ku2 ,k  +  V  k=0»l»  •  •  •  »N  (5.51) 

where  u  ,  u«  are  chosen  by  two  players  who  play  Nash  and  whose  respective 

X  3  K.  t  )  K 


costs  are  J^,^, 


J1  =  E[^+lQl,N+lXN+l  +  k!0(xkQl,kXk+ul>kUlk  +  U2,kRl,kU2,k)]  (5‘52) 
J2  =  EtxN+lQ2,N+lxN+l  +  k20^!CkQ2,kXk+u2,kl';,k+Ui,kR2,kUl,k)]  (5'53) 


where  Q,  .  =  o!  ,  > 0,  R.  =  r!  ,  > 0.  u.  ,  is  to  be  chosen  as  a  Borel  measur- 
l,k  i  ,k  i,k  i,k  i,k 


able  function  of  Z,  .  where 
k 


Zk  =  {yi,0’yi,l’"',yi,k} 


yi,k  Ci,kxk  +  Vi,k 
m,  m„ 


,  i  ■  1,2. 


(5.54) 

(5.55) 


x^,u^  k»^2  k  take  values  in  R  ,  R  ,  R  respectively,  w^,  v^  and  Xq  are 
independent  Gaussian  random  vectors,  the  expectations  in  (5 .43) , (5 .44)  are 
with  respect  to  all  random  variables  and  all  the  matrices  have  appropriate 
dimensions . 

Some  special  cases  of  this  game  have  been  treated 

successfully,  [17],  [63],  but  in  its  general  form,  is  quite 

difficult  to  solve.  Only  one  class  of  admissible  Nash  strategies  can  be 

relatively  easily  investigated  and  this  is  the  class  of  strategies  affine 

in  the  information.  If  player  1  uses  an  affine  function  oi  his  information 

23  at  each  stage  k,  i.e.  u.  ,  * M.  ,  zj  +  d..  ,  then  player  2  is  faced  with  a 

classical  LQG  problem  which  he  can  easily  solve  and  his  optimal  reaction 

2 

can  be  restricted  to  be  of  the  form  Uj  k  =  kzk  +  ^2  k  without  any 
deterioration  to  his  cost  The  converse  also  holds.  Thus  we  end  up  with 
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a  system  of  matrix  equations  with  unknowns  i=l,2,  k=0,...,N. 

This  system  might  have  one,  many,  or  no  solution  at  all.  The  following 
example  demonstrates  that  there  might  exist  no  solution  in  the  affine  class 
of  strategies. 

Example  5.2:  Consider  the  one  stage  game 


x-  =  Ax  +  B.  u..  +  B0u„ 

1  o  11  2  2 

=  E[x^Q^x^ +  u^u^ +  u^R^u^] 

J 2  =  Etx^C^x^  +  U2u2  +  U1^2U1  ^ 

where  x^ =  (x^.x^)'  has  zero  mean  and  covariance  equal  to  the  unit  matrix, 

2 

and  x.u^.u^  take  values  in  R  .  u^  and  u^  have  to  be  chosen  as  functions  of 
y^ * C^Xq  and  =  C2xo  resP®ctively .  Substituting  x^  with  its  equal  in 
J^,J2  We  oI:,cain  a  static  problem  with 


}1  =  E[U1 


J2  =  E[u^ 


I+B1Q1B1 


I+B2Q2B2 


The  solution  (u^,u2)  is  given  by 


,-l„ 


0 1 7  2 J 


2X22' 


.ww 

(5.56) 

(5.57) 

(5.58) 

‘1B2Q2B1E[u1ly2l. 

(5.59) 

Substitution  of  u^  from  (5.59)  into  (5.58)  we  obtain 


ux  -  -(I+B’Q1B1)"1B^QiAE[x(J|y1] 


+  (I+B’Q1B1)'1B’Q1B2(I+B’Q2B2)"iB'Q2AE[E[x0|y2] lyj 


+  (I+B^Q1B1)"1B’Q1B2(I+B'Q2B2)'iB2Q2B1E[E[u1|y2]|y1]. 
Let  us  now  choose 


,-l„ 


(5.60) 


w 
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Then  (5.60)  assumes  the  form 


which  is  impossible  to  solve  for  u^  if  6  ^  a. 

Since  the  information  of  each  player  is  affected  by  the  control 
laws  of  the  other,  one  would  be  tempted  to  consider  a  subclass  of  problems 
where  this  is  not  the  case.  Let  us  assume  that 


Ci,2[Bj,i:AlBj,0]  =  ° 

Ci,3[Bj ,2- A2Bj ,l*A2AlBj ,0]  =  0 


ci,ktBj,k-i:\-iBj,k-2:-*-:Ak-iV2-"AiBj,o]  =  0  (5-61) 


Cl,N[Bj,N-l-'AN-lBj,N-2-'"'AN-l’"AlBj,0]  ~  ° 
i^j,  i,j  =  l,2. 


Equation  (5.61)  means  that  the  controllability  subspace  of  player  j  is 
orthogonal  to  the  observability  subspace  of  the  C.  matrices  of  player  i. 

1  ,  K 

If  (5.61)  holds,  then  substitution  of  in  terms  of  o’‘'*’U1  k’ 

u^  q,...,U2  ^  for  k=0,...,N  (by  using  recursively  (5.51)),  in  the  cost 

functionals  wiii  result  in  a  Nash  game  of  the  type  ( 5 . 34 )— (5.37). 

The  L . ,  matrices  in  (5.30)  are  given  by  products  and  sums  of  the  C,  , 
ij  l,p 

A  ,B  (0<p<j)  matrices  and  the  H  matrices  by  products  of  the 
p  i ,  p  i 


A 
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C.,  ,A  ,  0<p<i  matrices.  Similarly  for  the  L...H.  matrices.  Of  course 
1,P  P  7  ij  i 

the  v.  ,  w,  noises  can  be  incorporated  together  with  x  in  a  random  vector 

1  j  K  K  U 

which  will  play  the  role  of  x  in  (5 . 34 ) — ( 5 . 37) .  Therefore  Theorem  5.6  can 
be  used  to  derive  conditions  which  guarantee  existence  and  uniqueness  of  a 
Nash  pair,  affine  in  the  information  for  the  game  (5 .51)— (5.55)  under  the 
hypothesis  that  (5.61)  holds.  Nonetheless,  since  for  the  Example  5.2 
the  condition  (5.61)  holds  trivially,  we  conclude  that  (5.61)  alone  does  not 
in  general  suffice  to  guarantee  existence  of  a  Nash  equilibrium  pair. 

Although  (5.61)  is  a  quite  restrictive  assumption  (since  for 
example  if  C  ,  =  I,  k=0,...,N  then  player's  2  controllability  subspace  is 

1 ,  K 

zero!)  it  did  not  really  make  our  problem  much  easier.  An  inspection  of  the 
static  problem  of  Section  5.2  shows  that  if  or  R2  (see  (5.2))  is  zero 

then  the  static  game  (5.1)-(5.2)  does  have  a  unique  solution  which  is  affine 
in  the  information.  Similarly  for  the  game  (5. 34)-(5 . 37)  if  the  R_'s  or 
the  R„'s  are  zero.  The  translation  of  these  conditions  to  problem  (5.51)- 
(5.55),  assuming  that  (5.61)  holds  is  that 


^Bi,k:AkBi,k-l;AkAk-lBi,k-2:  ' ' '  :W-1"  ,AlBi,0J  Qi,k+l^Bj  ,k:\Bj  ,k-l: 

;Ak"  'AlBj  V  =  °’  1=1  ’  i  =  Z’  k=0’1’--->N  (5- 


(5.62) 


holds.  Then  the  problem  of  player  1  is  totally  independent  of  u_  ,  i.e. 

Z  ,  K 

it  is  like  setting  R„  = 0  in  (5.38).  The  meaning  of  (5.62)  is  that  the 

observable,  through  Q  ,  parts  of  the  controllable  subspaces  of  players  1 

i ,  k 

and  2  are  orthogonal  in  a  "stagewise"  sense.  If  (5.61)  and  (5.62)  hold  then 
player  1  is  eventually  faced  with  solving  a  system  of  equations  like  (5.38) 


but  with  R  =0.  This  system  can  be  solved  uniquely  and  give  a  control  for 
player  u^  linear  in  the  information  under  invertibility  conditions  of  certain 
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matrices.  (The  reason  that  makes  these  invertibility  assumptions  necessary 
is  that  the  matrices  ^  arising  in  the  cross  terms 

might  make  the  control  problem  nonconvex  in  the  (u  n,u  ,  ...,u  ) 

variable.)  Consequently  player  2's  problem  can  now  be  solved,  since  it 
becomes  a  classical  LQG  problem.  For  solving  player  2’s  LQG  problem  no  new 
invertibility  assumptions  need  to  be  made.  Of  course  if  one  reverses  the 
roles  of  i=l,  j=2  in  (5.62),  similar  conclusions  hold  with  the  roles  of 
players  1  and  2  reversed. 

Other  cases  where  the  game  (5-51)— (5 .55)  can  be  solved  and  give 
solutions  linear  in  the  information  can  be  found  by  imposing  condition  (5.61) 
and  additional  conditions  which  guarantee  that  the  assumptions  of  Theorems 
5.6  and  5.7  hold.  We  will  not  carry  out  these  procedures  here  because  of 
the  complicated  character  of  the  conditions  involved. 


5.6.  Continuous  Time  Analogues 

In  this  section  we  will  deal  with  some  continuous  time  analogues 
of  some  cases  considered  for  the  discrete  dynamic  stochastic  Nash  game. 
Consider  the  system 

dx(t)  =  [A(t)x(t) +  B1(t)u1(t) +  B2(t)u2(t) ]dt  +  G(t)dw(t)  (5.63) 

and  the  cost  functionals 
Cf 

=  E [ /  [x'  (t)Q1(t)x(t)  +  u^(t)u1(t)  +  u2(t)R1(t)u2(t)  ]dt]  (5.64) 

^o 

Cf 

J2  =  E [ /  [x* (t)Q2(t)x(t)  +  u2(t)u2(t) +  u|(t)R2(t)u1(t) ]dt] .  (5.65) 

'  o 

£ 

u^  is  chosen  by  player  i  as  a  measurable  function  of  y^t={y^(s);  tQ<s<to), 


where 


and  u^.u^  are  such  chat  Che  solution  of  (5.63)  exists  and  is  unique  over 

(t  t,l.  x(t  )  =x  is  a  Gaussian  random  vector  with  given  mean  and  covariance 
o  f  o  o 

and  dWjdv^.dv^  are  standard  Wiener  processes.  xq , dw , dv^ , dv^  are 

independent.  The  initial  conditions  for  y  ,y^  are  zero-  x(t) .u^Ct) .y^Ct) 
m .  q . 

take  values  in  R  ,R  ,R  respectively  and  the  matrices  A.B^.Ck,  Q^  =  Q^>0, 
Ri=R^>0,  G,  R^=R^,  (R^R^O)  are  piecewise  continuous  functions  of  time 
with  appropriate  dimensions. 

The  case  in  the  discrete  problem  of  Section  5.5  where  each  player 
uses  an  affine  function  of  his  information,  corresponds  here  to  the  case 
where  each  player  uses 

t 

u  (t)  =  /  [d  n,  (t,s)  ]y  (s)  +  b  (t)  (5.67) 

X  £  S  -L  1  1 

o 

where  d  n.(t,s)  are  deterministic  Lebesque-Stielt j es  measures  and  b.(t) 
si  i 

deterministic  functions.  If  player  1  plays  u^  as  in  (5.67),  then  under 
appropriate  conditions  for  player  2's  problem  a  separation  principle  holds 
and  he  will  use  u^  of  the  form  (5.67),  see  [64],  Notice  that  (5.67)  is  a 
compact  form  of  writing  the  solution  of  an  infinite  dimensional  filtering 
problem.  The  need  of  infinite  dimensional  filtering  for  linear  quadratic 
continuous  Nash  games  was  first  pointed  out  in  [65].  The  study  of  possible 
nonlinear  solutions  to  the  game  of  this  section  is  far  more  difficult  than 
in  the  discrete  time  case. 

Let  us  try  now  to  translate  condition  (5.61)  to  the  continuous 
time  problem.  Let  $(t,t  )  be  the  transition  matrix  which  corresponds  to 
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Lemma  5.2:  If 


C1(t)$(t,x)B2(T)  =0  i#j, 

for  (i,j)  =  (1,2)  Chen  the  information  of  player  1  is  independent  of 
the  control  law  of  player  2. 

Proof :  From  (5.63)  and  (5.66)  using  (5.68)  we  obtain 
t  t  s 

y1(t)  =  /  C1(s)$(s,to)xodg  +  /  [/  ^(3)0(5, T)B1(x)u1(y1  T#x)dx]ds 


(5.68) 


t  t 
o  o 


t  S  L 

+  /  [/  C1(s)$(s,x)dw(x) ]ds  +  /  R1(s)dv1(s) 


(5.69) 


t  t 
o  o 


and  the  conclusion  follows. 


Obviously  (5.68)  is  the  analogue  of  (5.61)  for  i=l,j=2. 

Let  us  now  find  "the  analogue  of  (5.62).  The  way  we  derived  (5.62) 

was  by  imposing  the  condition  that  there  are  no  cross  terms  u|  ^A^  ^ 

0<k,  £<N  (A:  some  matrix)  in  J-^.  Therefore  here  we  demand  that 
t  t 

(/  $(t,x)B(x)u. (x)dx) ' Q. (t) (/  $(t,x)B  (x)u  (x)dx)  =  0  (5.70) 

t  t 

o  o 

for  every  t  and  every  t^.u^  (5.70)  can  be  written  equivalently 

t  t 

/  /  u' (x)B' (x)$' (t,x)Q  (t)$(t,o)B 2(o)u2(o)dxdo  =  0 

t  t 
o  o 

and  thus  we  conclude  that  (5.70)  is  equivalent  to 


b' (x)4>' (t,x)Q  (t)4>(t,o)B  (0)  =0,  t  <o,x<t  te[t  ,t  ]  (5.71) 

1  1  L  o  or 

(5.71)  is  clearly  the  continuous  time  analogue  of  (5.62).  We  can  now  state 

the  following  theorem. 

Theorem  5.8:  Assume  that  (5.68)  holds  with  (i,j)=  (1,2)  and  (i,j)=  (2,1) 
and  that  (5.71)  holds.  Then  for  the  Nash  game  (5.63)-(5.66)  player  1  can 
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find  his  control  by  solving  a  control  problem  with  state  y^(t)  obeying 
(5.69)  and  a  quadratic  cost  depending  only  on  and  y  . 

Clearly  the  quadratic  cost  mentioned  in  Theorem  5.8  will  be  of 
a  more  general  form  than  the  familiar  one,  and  the  control  problem  of  player  1 
will  be  of  the  category  considered  in  [64].  The  results  of  [64]  can  be 
used  for  solving  the  control  problem  of  player  1  under  the  assumptions  of 
Theorem  5.8.  The  invertibility  type  conditions  for  solvability  of  this 
problem  in  the  discrete  time  case  (see  discussion  after  (62))  have  to  be 
made  here  too.  As  soon  as  this  control  problem  is  solved  player  2  will 
be  faced  with  a  linear  quadratic  problem  of  the  form  treated  in  [64].  Thus 
we  conclude  that  the  assumptions  of  Theorem  5.8  in  conjunction  with  the 
results  of  [64],  can  lead  with  relative  ease  to  the  solution  of  the  problem. 

We  will  close  this  section  with  a  particularly  simple  case  for 
which  the  Nash  game  can  be  solved  without  using  the  results  of  [64]  on  the 
infinite  dimensional  stochastic  regulator  problem.  Let  us  assume  that 

C_(t)<f(t,t  )B  (t)  5  0 
2  o  1 

C1(t)<t(t,tQ)B2(t)  =  0  te[tQ,tf]  (5.72) 

CL(t)  =  C1(t)Q^  (t),  Qx(to)  =  I. 

"2 

Let  us  also  assume  that  (t)  is  continuously  differentiable  and  invertible 
on  [t  ,tf].  Let 

1/2 

z(t)  =  Q1  (t)x(t).  (5.73) 

From  (5.63)  and  (5 .72  )  ,  (5 .73  )  we  obtain 

1/2  t  VI  t  l/i 

z(t)  =  (t)$  (t,tQ)xQ  +  /  Q1  (t)$(t,i)B1(i)u1(T)dT  +  /  Q1  (t)4>(t,x)G  (x)dt. 

Co  Co  (5.74) 

From  (5.66)  and  (5.72)  we  obtain 


131 


dy^(t)  =  C1(t)x(t)dt  +  R1(t)dv1(t)  =  (^(OQ^  (t)x(t)dt  +  R^tjdv^t) 
=  C1(t)z(t)dt  +  R1(t)dv1(t). 


Let 


w 


Ul 


_>>2 


A(t)  =  [Q1(t)A(t)  +~  Q1(t)]Q1  (t) 


(5.75) 

(5.76) 


Then  player  l's  problem  can  be  written 


dz(t)  =  (A(t)z(t) +  B^(t)u^(t) ]dt  +  G(t)dw(t), 


z(t  )  =  x 
o  o 

dy1(t)  =  C1(t)z(t)dt  +  R1(t)dv1(t) 


(5.77) 


mini"  (z' (t)z(t) +  u|(t)u^(t))dt 
Co 

where  we  omitted  the  term  u^^R^Ou^t)  since  the  first  two  of  the 
assumptions  (5.72)  guarantee  that  y2(t)  is  independent  of  u^,  (see  Lemma  5.2). 
(5.77)  is  a  classical  stochastic  linear  regulator  problem.  Player  1  has  to 
use  E[z(t)|y  ]  *  z(t)  and  his  control  law  is  linear.  Player  2  will  be 

1  ,  L 

faced  with  a  classical  linear  quadratic  control  problem  with  state 

(x(t),z(t))  and  he  will  have  to  calculate  E[x(t)|y„  ]  and  E[z(t)|y„  ] 

*•  >  t  z ,  t 

i  n 

=  Q  (t) E[E[x(t) | y  ] J y  ];  i.e.  player  2  will  use  a  two  step  filtering 

1  ,  L  L  y  t 

procedure.  We  thus  see  that  conditions  (5.72)  single  out  a  class  of  problem 
with  nonnested  y^  and  y2  which  admits  a  unique  solution  linear  in  the 
estimates  E[x(t)|y1  t],  E[x(t)|y2  ,  E[E[x(t)|y1  ] | y2  tJ  and  there  is  no 

need  for  infinite  dimensional  filtering. 
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CHAPTER  6 
CONCLUSIONS 

The  work  presented  in  this  thesis  answers  certain  questions  in  the 
area  of  Leader-Follower  and  Nash  differential  games.  The  main  topics  include 
existence  of  closed-loop  Nash  strategies,  necessary  conditions  for  closed- 
loop  Leader-Follower  strategies,  sufficient  conditions  for  Leader-Follower  and 
Nash  strategies  with  memory  and  stochastic  Nash  games  with  state  information. 

In  Chapter  2  we  gave  conditions  which  guarantee  the  existence  of  linear 
closed  loop  Nash  strategies.  We  also  established,  in  this  chapter,  the 
uniqueness  of  analytic  closed  loop  Nash  strategies  for  analytic  differential 
games.  In  Chapter  3  we  gave  necessary  conditions  for  closed- loop  Leader- 
Follower  strategies.  It  was  shown  that  the  leader's  problem  is  a  singular 
control  problem,  which  explains  the  peculiar  behavior  of  certain  previously 
studied  examples.  It  was  also  established  that  the  leader  can  in  general 
restrict  himself  to  strategies  affine  in  the  state  and  that  there  are  other 
types  of  strategies  to  which  the  leader  can  restrict  himself  without 
deteriorating  his  cost.  These  different  types  of  strategies  although 
guaranteeing  the  same  optimal  costs  and  trajectories,  might  result  in  different 
sensitivity  or  stability  properties.  It  was  also  shown  in  this  chapter  that 
the  Principle  of  Optimality  holds  for  Leader-Follower  differential  games,  if 
and  only  if  it  is  in  the  players'  common  interest  to  consider  the  leader's 
problem  as  a  team  problem.  In  Chapter  4  we  dealt  with  sufficient  conditions 
for  Leader-Follower  and  Nash  differential  games  where  the  players  have  recall 
of  previous  values  of  the  trajectory.  In  particular  for  the  Leader-Follower 
game  we  considered  the  case  where  the  Leader's  problem  is  treated  as  a  team 
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problem  by  both  players.  The  conditions  of  this  chapter  were  given  in  the 
form  of  integral  equations.  In  Chapter  5  we  considered  stochastic  Nash 
games.  Results  concerning  the  uniqueness  of  linear  strategies  were  derived 
for  the  static  case  and  translated  to  the  dynamic  one.  For  the  dynamic  case 
it  was  shown  that  if  the  control  of  one  player  does  not  affect  the  information 
of  the  other  and  if  at  least  one  player's  cost  is  not  affected  by  the  other 
through  the  penalization  of  the  state,  then  under  invertibility  conditions  for 
certain  matrices,  the  linear  solution  is  the  unique  one. 

There  is  a  plethora  of  problems  to  be  studied  in  the  area  of 
differential  games  with  state  information,  besides  those  studied  in  this 
thesis.  We  will  single  out  some  of  them  arising  naturally  from  the  study 
presented  here.  Chapter  .2  suggests  the  following  areas  of  research: 

1)  Uniqueness  (or  nonuniqueness)  of  closed  loop  Nash  strategies.  2)  Geometric 
conditions  for  solvability  of  the  coupled  Riccati  equations.  3)  Efficient 
algorithms  for  solving  the  coupled  Riccati  equations.  Chapter  3  suggests  the 
following  areas  of  research:  1)  Generalize  the  analysis  in  Chapter  3  to 
nonstandard  control  problems  where  x^.  *  {x(@)  |  0<  @  <  t}  plays  the  role  of  x(t). 

2)  Generalize  the  analysis  in  Chapter  3  to  L-F  games  where  the  leader  uses 
u(xt,t).  3)  Efficient  algorithms  for  solving  the  L-F  game  of  Chapter  3. 

Chapter  4  suggests  the  following  areas  of  research:  1)  Existence  and  unique¬ 
ness  of  the  solutions  presented  in  Chapter  4.  2)  Single  out  simple  classes  of 

L-F  and  Nash  games  with  memory  and  provide  computational  procedures.  Chapter  3 
suggests  the  following  areas  of  research:  1)  In  view  of  the  possible  nonlinear 
solutions  to  the  games  of  this  chapter,  provide  simple  sufficient  conditions  to 
check  optimality  of  nonlinear  solutions.  2)  Geometric  interpretations  of  the 
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matrix-type  conditions  of  this  chapter.  Of  course  in  all  chapters,  questions 
of  sensitivity  and  stability  character  of  the  solutions  arise  especially  when 
the  solutions  of  the  games  are  not  unique. 
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APPENDIX  A 

PROOF  OF  PROPOSITION  A.l 

In  [4],  Proposition  1  states  that  given  (2.1), (2. 2)  where 
Rll’^22  >  then  if  K^,  satisfying  (2 . 4)  exist  and  A  (as  in  (2.5))  is 

A.S.,  then  the  strategies  (2.7)  satisfy  (2.3).  This  is  not  true  as  the 

.  2  2  2 
counterexample  x  =  x  +  u  +  v,  -  =  J  q  (x  +u  “  2v  )dt  demonstrates. 

[This  example  is  used  in  [9]  to  show  that  in  the  zero  sum  case  the  linear 

solution  for  the  game  over  a  finite  period  of  time  [0,T]  does  not,  as 

T  -*  +®,  tend  to  the  linear  solution  of  the  infinite  time  case.]  But  if  one 

makes  additional  assumptions  then  the  conclusion  holds. 

The  correct  form  of  Proposition  1  in  [4]  is 

Proposition  A.l:  Given  the  system  (2.1)  and  the  two  functionals  (2.2)  where 

Qi>  R^  are  real  symmetric  matrices  and  R^^>R22  >  there  exist  real 

symmetric  matrices  K^,  satisfying  (2.4)  and  (2.5)  and  either  (i)  or  (ii) 

hold : 

(i)  0.  +  K.rTI'R.  .rT1K.  2:  0,  i,  j  =  1,2,  i  t  j. 

J  JJ  ij  jj  ] 

(ii)  The  two  control  problems 

x  =  (A  -  B.rT1B'.K.)x  +  B.u. 

J  JJ  J  J  13- 

+®  -1-1 

min  Jo  (x'  [Qi+  KjRjjRijRj  jKj^x  +  uiRiiui)dt’  1  *  J’  =  1>2 

satisfy  the  conditions  of  Theorem  2,  p.  167  in  [14],  then  the  strategies 

•k  k 

(2.7)  satisfy  (2.3)  and  are  finite. 
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APPENDIX  B 
NONFINITE  J*'s 


The  case  where  at  least  one  of  J^,  is  could  also  be 
examined.  For  example,  if  we  are  interested  in  a  linear  Nash  equilibrium 
where  =  +=°  and  J2  is  finite,  then  this  amounts  to  seeking  ^  -  -I^x, 
i  =  1,2  where  (i),  (ii),  (iii)  hold: 

(i)  The  control  problem 

x  »  (A  -  B1L1)x  +  B2u2 

_  +tt>  (  ^ 

J2  =  min  Jo  (x'(Q2  +  Li  R21L1)x  +  U2R22U2)dt 

has  J2  (=  J2)  finite.  (For  example  assume  controllability  and 

(ii)  The  problem 

x  =  (A  -  b2l2)x  +  Blul 
^00 

J1  ■  Io<*'(VL2h2L2)X+U;RllVd,; 


has  =  +»  for  every  u^  =  -L^x  (which  means  roughly  that  some  uncontrollable 

mode  \  of  the  pair  (A-  B2L2’Bi^  which  does  not  lie  in  the  nul1  sPace  of 

Qx  +  L2Ri2L2  ^as  Re  ^  2  0)* 

(iii) 


R22B2K2 


K2(A-  B^*)  +  (A  -  B^L*)  ' K2  +  (Q2+L*'R21L*)  -  V2R22B2K2  =  0. 


*  *  * 

Similarly  one  can  form  conditions  for  the  cases  J ^  =  J2  =  +=°,  =  *®  and 

it 

J2  finite. 
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APPENDIX  C 
PROOF  OP  2.32 


Consider  the  matrix  differential  equation 


2  2 


K1S1K1 

"  K1S2K2  ‘ 

K2S2K1  + 

K2SolK2 

(C.l) 

K-S-K. 

-  K.  S ,  K,  - 

K,  S,K„  + 

K,  S  „K_ 

CC.2) 

2  2  2 

2  1  1 

11  2 

1  o2  1 

where  K.^,  K2  are  time  varying,  t  a  0  and  K^O)  =  T ^ ,  K2(0)  =  F2  are  the 
initial  conditions.  Then  it  follows  that  f or  t a  0  sufficiently  small  it  holds 

t 


KL(t)  =  eAtT1eA  C  +  J'oeAc[Q1  -  K1S1K1  -  K^K,,]  -eA  CTda 


A'a 


(and  a  similar  one  for  K2(t)).  The  constant  matrices  P2  solve  (2.67), 
(2.68)  if  and  only  if 


rL  -  +  J0eAa[Q1  -r1s1r1  -r^ -r2s2r1  +  r2solr2]eA'CTda 

(and  a  similar  one  for  r2).  Because  A  is  A.S.  and  1"^,  ?2  constant,  the  integral 


+®  , 

I.  ■  .foei°[<>l-riSiri-r1S2r2-r2S2ri  +  r2Soir2le  dx 

exists.  Also 

;Vt^-rlVl-W2 -r2s2r1  +  r2s0lr2]eA'ada  - 

(w  =  a  -  t ) 

=  Jo  eA(wf  )[Q1  -riSiri  2  i  +  dw  = 

_  AtT  A't 
=  e  I  e 

CO 
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and  thus 


t 

r  + 


u  o 


AtT  A't 
e  I  e 

00 


from  which  we  conclude  that  I  =  F, .  A  similar  result  holds  for 

CO  1  Z 

ducing  scaling  F^  *  aF,,,  a  >  ^  and  setting  =  F i>  we  have  (2.32). 


Int 
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APPENDIX  D 
PROOF  OF  2.36 


It  is  easy  to  see  that  £  ]!t||  | j T  ^||  |'e’  *■!] .  Let  t  >  0  and 


So,  if  x  ^  ^  ^  ^  ^2*  *  •  •  »x^  ) 


R 


sup  -I A-f^Sj  =  sup  {j|x|!  +  tl[A  .xjl]  = 
!x|i=l  |!x!!  =  l  .  ' 


=  1  +  t  sup  ;'A-x||  =  1  +  t  sup  i|A.x|! 

1  1 


+  t;|A 


and  thus 

!|Ai!i  £  i  +  t(i  +  |  i'Ajj). 


Continuing  similarly  we  end  up  with 


ISJ  |rr 
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|a.||  *  1  +  t(i  +  |(i  + 1(. ..  ((i  +  ttj)  (i  +  irr))' 

_  i  ,  t  t2  t1'1 

- 1  +  -+  ir+  ••• +  • 


.)))  = 


Therefore,  if  m  =  max (m^, . . . ,m  ) 

Ftn  ^  n_mi__l|||i  Atu  n„im„*lii  ^t 


,  .  .  ,  .  .  m- 1  m- 1  i 

Ti|  ||T  l||le  '!  *  Ht'I'It  ';le  t(l  +  Tr+  ...  +"A^V,)  =  p  I  (rr) . 

i-  (m-1).  r,_0  j. 


Direct  calculation  gives  (recall  X  <  0) 


+°°  +<»  —  m-1  i 

r  S!eFt,, 2__  ^  2  r  2>.tr  „\tJ.,2 

J  n 


£  z  p  ikt  z  -  2  r  -1  2Xt  t  +J  j 

dt  *  P  Jo  e  [  2  C-T)]  dt  -  p  Jo  a.  e  --T-rdt  = 

j=0  J  i,j=0  ‘J 


■  p!  T  iiiiii  f-ui+J+1 . 

i,j=0  1,J’  \-2\l 


P  n(-lA), 


CD-I) 


T 
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APPENDIX  E 

SUFFICIENT  CONDITIONS  FOR  ASSUMPTION  (A) 


In  this  Appendix  we  give  certain  conditions  under  which  Assumption 
(A)  (Section  3.2)  holds. 

Lemma  E . 1 :  Let  U  be  a  subset  of  U  (see  (3.1))  ,  defined  as 

Aj 


U  =  £u£U|u(x,t)  =  C(t)x  +  D(t),  where  the  m.  /n  matrix  C(  ) 

Ar 

and  the  x  1  vector  D(t)  are  piecewise  continuous  (E.l) 

functions  of  time  over  ft  , t-]t. 

o  r  1 


Then  it  holds  : 


inf  J .(u,v)  &  inf  J  (u,v)  a  inf  J  (u,v)  *  inf  J  (u,v) 

1  (E .  2 ) 

uSU^vCTu  u€U,  v«Tu  u(U,  vST'u  uSU^  v(l'u. 

Proof  :  The  inequalities  follow  from  the  facts  U  c  U,  TucT’u  yu€u.  The 

l  ~  ~ 

last  equality  is  obvious  in  the  light  of  (3.26)  and  the  proof  of  Theorem  3.2. 


An  immediate  conclusion  of  Lemma  E.l  is  that  if 


inf  JL(u,v)  *  inf  J^(u,v) 
u£  Ux,  v?  Tu  u£  Ur  v€  T’  u 


(E.3) 


holds,  then  Assumption  (A)  holds  (with  U  =  U).  For  (E.3)  to  hold,  it 
suffices  that  the  first  order  necessary  conditions  for  the  follower's 


problem  are  also  sufficient,  for  each  fixed  uf  U  .  More  specifically,  for 

2 

fixed  C(t),  D(t)  as  in  definition  (E.l)  ,  we  consider  the  problem 


j 


(E.4) 


minimize  h(x(t  ))  +  f  M(x,C(t)x+ D(t),v,  t)dt 
£  ’  t 

o 

subject  to :  v  6  V 

X  =  f  (x,C(t)x  +  D(t),v,t),  x(tQ)  =  xo>  t6[t0,tf] 

and  seek  conditions  under  which  the  first  order  necessary  conditions  for  an 
optimal  v*  for  problem  (e.4)  (see  (3.10-2)-(3.10-4))  are  also  sufficient.  Such 
conditions  can  be  found  in  Chapter  5-2  of  [39]*  We  formalize  this  discussion 
in  the  following  Proposition. 

Proposition  E.l:  If  for  each  uSU^,  the  first  order  necessary  conditions 
(3. 10-2) -(3. 10-4)  for  problem  (E.4)  are  also  sufficient,  then  Assumption  (A) 

holds . 

The  discussion  in  the  present  Appendix  generalizes  clearly  to  the 

case  where  each  u1  depends  on  h1(x,t)  instead  of  x  and  to  the  case  where 

different  U  ' s  are  considered;  see  for  example  Proposition  3.1(ii). 

JO 

As  an  example  where  Proposition  E.l  can  be  applied,  we  consider 
the  linear  quadratic  game  of  Section  3.  Then,  Theorem  5,  p.  341 

of  [39]  in  conjunction  with  Proposition  E.l  yield  that  if  2  0,  ^  0, 

r  <£  0,  2  o  then  Assumption  (A)  holds. 
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APPENDIX  F 
PROOF  OF  THEOREM  3.1 


Proof  of  Theorem  3.1:  Let  g=0  w.l.o.g.  (see  [35]).  Consider  a  function 
9  €  U,  9  =  (cp 1 ,  . . .  ,cpm)  which  has  the  same  continuity  and  differentiability 
properties  as  u*.  Such  a  cp  will  be  called  admissible.  Using  the  known 
theorems  on  the  dependence  of  solutions  of  differential  equations  on 
parameters,  we  conclude  that  for  s  €  R,  s  sufficiently  small,  u*+eo  gives 
rise  to  a  trajectory  {  (x (e , t) , t) 1 1 € [t  , t f 1 } ,  x(0,t)  =  x*(t),  and  that 
x(c,t)  is  in  w.r.  to  s.  Direct  calculation  yields 


4r  =  £  +  (u  +ecp  )f  +  £  (UL  4-ecp1  )f.  ]' 

dt  be  '  ^x  ^  x  9c'  u  £=]/  xx  'xx'  i J  oe 


We  set 


m 


+  f'9  +  .Z1f.'V  hV-  ,  2^ 

uT  i=i  i  x  T  i  ’  a 


Lslfii.t) 


t=t 


=  0. 


(F.l) 


./t\  =  ^x(e,t) 

Se  e=0 


m 


V‘>  -  ^ 

B2(t)  =  fiVxhL}  i  = 


(F.2) 


A(t)  =  f  +  u  f  +  Z  [7  hV.  .7  h1'+Zu17  tuf .]  p  (F.3) 

X  XU  i  =  1L  X  11  X  j=l  J  XX  j  1 

(F.4) 
(F.5) 


where  A,  B^,  B2  are  evaluated  at  t,  x*,  u  ,  u“  and,  thus,  for  e=0,  (F.l)  can 
be  written  as 


m 


z  =  Az  +B.c?  +  _  Z  B^cp1  i»  z  (t  )  =  0. 

i-1  y  ® 


(F.6) 


For  fixed  c 3  we  consider 


J(€)  =  J(u  +  59). 
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—  1  * 

Since  j(e)  is  in  C  w.r.  to  e  and  u'  is  a  local  optimum,  it  must  hold 


dj(e) 

de 


e=0 


=  0. 


Direct  calculation  yields 
t 


-  J*  [[l  +  (u  +ecp  )L  +  £  (u^  +eo^  )L.] 
de  L  x  x  x'  u  i=i  xx  xx  l 


+  L  cp  +  .^.L.'v  h\  Jdt 
u  i=l  1  x 


5x  (e  ,  t ) 


de 


(F.  7) 


Setting 


m  .  .  .  ,  ^i 

F  (t)  =  L  +u  L  +  2  [V  hlul.  .V  h1  +.z,u37  h^]  L. 

x  x  u  i=iL  x  11  x  j=l  j  xx  j  1 


y  y 


A.  (t)  =  L 
l  '  u 


A2  ^  ~  LiVxh  ’  1  1> • • • >m- 


(F.8) 

(F.9) 

(F.10) 


with  T,  A^,  A2  evaluated  at  x  ,  u*,  u*,  we  conclude  from  (F.7).(F.l0)  that 
te 


■E  m  .  . 

n  [Tz  +A,cp+  E  A^C31.]dt  =  0. 
1  1  i=l  2  i 

t  y 


(F.ll) 


Therefore  (F.ll)  must  hold  for  every  admissible  cp.  Let  $(t,T)  be  the  transition 
matrix  of  A(t).  Let  also  cp  ( t)  denote  the  vector  (cp1  (h1  (x*  (t) ,  t)  ,  t) , .  . . , 
cpm(hm(x*(t)  ,t)  ,t))  '  and  ^(t)  the  vector  (h'  (x^t)  ;  c)  a11).  .  Then  from 
( F .6  )  we  obtain 


Z(t)  =J  $  (t.Tj^B^T^CT)  +  .E1B2(T)Cp1(T)]dT 


(F. 12) 


^[to.tf] 


and  substituting  in(F.ll)we  obtain 
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P  r>  —  m  i  _i 

J  {r(t)J  #(t,T)[B1(T)<P(T)  +  ,IlB2(T)Cp(T)]dT  +  Aj(t)5(t) 


m  . 

+  I  A0(t)q>  (t)}dt  =  0. 
i=l  * 


(F .13) 


k[a  b]  denote  the  indicator  function  of  [a,b]  c  [tQ>tf]  .  We  can  inter- 


Let  Xj 

change  the  order  of  integration  in(F.13)  since  the  integrated  quantities  are 
bounded  on  [tQ,t^]  x  [t^t^]  (Fubini's  Theorem).  Using  the  fact  X(c)  = 


[c,tf] 


X(b)  we  have  successively 

tf  tf  m  . 

/  J  [r(t)§(t,T)B.(T)cp(T)  +r(t)S(t,T)  E  B^(T)^(T)] 
t  t  1  i*l  2 

o  o 

tf  fcf 

*  X(t)  dTdt  *  f  [J  r(t)$(t,T)dt]B  (t)9(t)  dT  + 
fT.tJ  "'t  T 


m  t,.  t, 
m  f  f 


+  z  f  [J  r(t)« (t,T)dt]Bj':T)9,'(T)dT, 

i-r  t  t  1 

o 


By  introducing 


p' (t)  =  r  r(t)5(t,r)dT 

T 


(F..13)  can  be  written  as 
t. 


m 


J  Ip'(t)B1(t)  +  A^t)]«P(t)  +  .ZL  J  1P'(t)B2(t)  +aJ(t)] 


^(t)  dT  =  0. 


Applying  Lemma  3-1  to  (f.16),  we  obtain 


p'(t)Bl(t)  +A1(t)  s  0,  on  ltQ,tf] 


[t0»b] 


(F.14) 


(F.15) 


(F.16) 


(F. 17) 


Using  (F.4),  (F.5)  and  (F.9),  (F.10)  in  (F.17),  (F.18)  we  have  equivalently 
(3.20)  and  (3.21).  Differentiation  of  (F.15)  and  use  of  (F.3)  and  (F.8)  give 
the  equivalent  to  (F.15) 


P(tf)  =  0. 


The  assumption  g=0,  is  removed  in  the  known  wav,  resulting  in  (3.22). 
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APPENDIX  G 
PROOF  OF  THEOREM  4.1 

Proof  of  Theorem  4.1:  Consider  the  functions 


‘1 

*2 

*3 


X 

c 

X 

n 

®,m 

1  X 

C 

X 

L 

n 

®,  m 

L  X 

C 

X 

L 

n 

oo, m 

1  X 

C 

X 

L 

n 

®, m 

U) 

€  1 

Rn 

*  c„ 

(G.l) 


H^(?  ,X,U)  (t)  =x(t)  -  I  A(T)x(T)dT  -  J  B(T)u(T)dT.X( 


■-0 

t, 


H2(?,x,u)(t)  =  j*  [d  Tl(t,s)]x(8)  +  J*  tdg111(t,s)]u(s)  -q(t) 


H3(?,X,U)  -  §  -  XQ  -  J  A(T)x(T)dT  -  J  B(T)u(T)dT 


..  r 

J(?,x,u)  -  7  [5'F  ?  +  j*  (x'  (t)Q (t)x (t)  +  u'  (t)R(t)u(t) )dt] 


(G.2) 


Clearly,  ,  H^,  and  J  are  well  defined.  To  show  that  H2  is  well  defined 

Cf 

it  suffices  to  show  that  if  u£L  then  J  [d  (t,s)]u(s)  €l  ,  • 

»,tn  ^  si 

o 

Let  u  €  l  ,  i|ullT  ■  M.  Then  there  exists  a 
®,tn  Ha 

sequence  fu  3  of  continuous  functions  u  :  [tn,t  J  -R  ,  ^ :  u  (t)  -  u(t) 

"  n-l  nut  n 

a.e.  and  ) u^(t)  1  s  M  +  1  V  t  €  [tQ,tf] ,  Yn,  (see  Theorem  3,  page  106  in 
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[49].  y  (t)  =  f  [d  T|  (t,s)]u  (s)  is  measurable,  |y  (t)|  s  (M+l)m  (t) 
n  v  s  1  n  n  1 

and  thus  y  € L,  .  Since  u  -  u  a.e.,  by  Egoroff's  Theorem  we  have  that 
Jn  l,m  n  J  ° 

V  e  >  0,  -*  0  as  n^  +•*  where  A^  =  [s  :  s  €  [  tQ,t ^]  ,  |  un(s)-u(s)|  >e]  . 

It  holds 


<  e-c1(t)  +  (2M+l)c1(t)nje(A^). 

Cf 

Since  is  finite  a.e.,  letting  n-*+®  we  obtain  |limyn(t)-J’  [  dgn^  (t  ,s )]  u  (s)  | 
<  e-c^(t),  a.e.  in  t€[tQ,t^]  where  lim  yn(t)  stands  for  either  limsup  or 
liminf.  Since  this  inequality  holds  Ve>0  we  conclude  that 
tf 

J*  [dsT1(t,s)]u(s)  =  lim  yn(t>  a.e.  in  U0,tf]  •  (G.3) 

Since  |yn(t)l  <  (M+l)c^(t)  and  (A  3)  holds,  we  conclude  by  Lebesgues  theorem 

that  fcf 

J*  [ds'ni<t,s)]u(s)  €  Lj  k- 

*"0 

Problem  (P)  can  be  written  equivalently 


minimize  J(f,x,u) 

subject  to  Hi(§,x,u)=0,  i  =  1,2,3 

(§,x,u)€Rnx  Cnx  -  P. 


(G.4) 


By  Theorem  1,  page  220  in  [50]  we  conclude  that  a  sufficient  condition  for 

(?*,x*,u*)  to  solve  (g.4)  is  the  existence  of  a  (P,\,k)  €  (C*,L..  ,  ,Rn)*such  that 

n  1»k 


denotes  Lebesgue  measure  on  [t  ,t,]  . 
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J(S*,x*,u*)  +<H1(?*,x*)u’9,fi>  +<H2(?*,x*,u*)A>  +<H3(§*,x*,u*),k) 

<  J(oo)  +(H1Cn),li>  +<H2(u)),X>  +  <H3(ffl),k>  y<u<=n.  (G.5) 


Since  the  function  J(tt>)  =  J(tu)  4-  (u>)  ,g.)  4- (h^  (w)  +  (Ej  (uu)  ,k)  is  convex 

and  Frechet  differentiable  a  necessary  and  sufficient  condition  for  (G.5)  to 
hold  is  that 

dJ (5* ,x* , u* ;  C,h,v)  =  0 

(G.  6) 

V(S1h)v)6R  xcnuv 

where  dJ  denotes  the  Frechet  differential.  Straightforward  calculations  result 
in  the  following  explicit  form  for  (G.6). 


(§  1 F  +  If ' )  C  *  0  V?€Rn 


(G.7) 


f  J  J  f 

j  x'  (t)Q(t)h(t)dt+J  [dp. '  (t)]h(t)  +j  \'(t)([  td  Tl(t,s)]h(s))dt 

t  t  t  t  • 

o  o  0  0 

t  t 

pf  ,  pf 

+J  M-  (t)A(t)h(t  )dt  -k  J  A(t)h(t)dt  *  0  Vh  €  C  (G.8) 
t  t  n 


l-  *■  J.  J- 

u'  (t)R(t)v(t)dt  +J  |i  '  (t)B(t)v(t)dt  +/  \'(t)  (f  [dsV1(t,s)]v(s))dc 


-k'J  B  (t  )v  (t  )dt  =  0,  Vv€L 
t 


LG. 9) 


Use  of  the  unsymmetric  Fubini  theorem  in  [47]  yields 


(G. 10) 


(G. 11) 


Using  (G. 10) , (G. 11)  in  (G.8),(G.9)  we  obtain  the  sufficiency  conditions  (4.41), 
(4.42)  wherein  we  substituted  g.  by  u-k,  and  k  by  -Ff *  =  -Fx(t^).  D 
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